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Foreword 


In  God  we  trust;  all  others  must  bring  data. 

—Attributed  to  W.  Edwards  Deming 

While  the  United  States  achieves  its  national  security  missions  by  equipping  its  military  forces 
with  the  best  weapons  systems  in  the  world,  questions  continue  about  the  performance  of  the 
defense  acquisition  system.  How  effective  is  it?  How  can  that  effectiveness  be  objectively 
measured?  Can  we  use  those  measures  to  affect  behaviors  with  appropriate  incentives  or 
determine  which  policies  and  procedures  improve  results  and  which  turn  out  to  be  misguided? 

Answering  these  questions  requires  more  than  opinion.  It  requires  analysis  of  unbiased  data  to 
discover  insights  into  underlying  effects.  These,  in  turn,  will  inform  better  policy  and 
programmatic  decisions. 

This  is  the  first  in  a  series  of  planned  annual  reports  on  the  performance  of  the  defense 
acquisition  system— its  programs,  institutions,  workforce,  managers,  executives,  and  industrial 
partners.  By  using  objective  data  and  analysis  to  measure  performance,  these  reports  will 
identify  underlying  drivers  and  inform  future  decisions  on  programs,  policies,  and  processes. 

This  first  report  focuses  primarily  on  performance  related  to  Major  Defense  Acquisition 
Programs1  (MDAPs).  It  will  not  delve  into  specifics  of  the  individual  programs  but  is  intended  to 
use  aggregated  data  from  these  programs  to  shed  light  on  macro-level  performance  indicators 
for  the  acquisition  system  as  a  whole.  The  report  focuses  on  more  in-depth  indicators  of  system 
outcomes,  particularly  with  respect  to  cost  and  schedule,  and  does  so  by  looking  at  various 
institutional  trends. 

•  How  does  the  portfolio  of  major  programs  perform  over  time?  What  has  changed,  how,  and 
by  how  much? 

•  Are  there  differences  associated  with  leadership? 

•  Are  there  differences  among  DoD  organizations? 

•  Are  there  differences  among  our  industrial  partners? 

This  report  also  reflects  results  to  date  from  the  compliance  of  the  Department  of  Defense 
(DoD)  with  the  Improve  Acquisition  Act  of  2010  (Title  VIII,  Subtitle  F,  Public  Law  111-383, 
Section  861  codified  in  10  U.S.C.  §  2548)  on  performance  assessments  of  the  defense 


1 10  U.S.C.  §  2430. 
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acquisition  system.  While  similarly  motivated,  our  efforts  will  go  beyond  the  specifics  of  this  act 
to  seek  additional  insights  for  improving  the  performance  of  the  defense  acquisition  system. 

In  addition,  this  study  will  be  used  in  part  to  fulfill  a  recent  request  from  the  Office  of 
Management  and  Budget  for  an  evidence-based  analytic  study  on  acquisition  performance. 

Readily  available  data  allowed  us  to  provide  historical  baselines  on  acquisition  performance  and 
some  initial  insights  into  whether  performance  has,  or  has  not,  improved  recently.  They  also 
demonstrate  that  it  can  take  many  years  to  see  the  results  of  new  policies,  making  it  even  more 
important  to  test  and  inform  those  policies.  Although  existing  data  can  be  effectively  leveraged 
to  improve  our  understanding,  a  lesson  learned  is  that  gaps  remain;  therefore,  I  initiated  a 
strategic  initiative  to  identify  those  key  data  gaps  and  begin  selective  collection  of  new  data  for 
future  analysis.  That  work  will  continue,  and  will  inform  future  reports. 

Since  this  initial  report  focuses  primarily  on  analyzing  MDAP  development  and  early  production 
information,  it  cannot  be  considered  a  complete  picture  of  the  entire  acquisition  system.  Future 
reports  will  delve  into  areas  such  as  contracting,  acquisition  of  services,  technology 
development,  industrial  base  concerns,  etc. 

Value  obtained  in  acquisition  is  a  balance  of  costs,  benefits,  and  prudent  risks.  Risks  are  a  fact  of 
life  in  acquiring  the  kinds  of  products  our  warfighters  need,  and  these  risks  must  be  objectively 
managed.  Additionally,  demands  and  threats  do  change  in  both  the  short  and  long  term,  so  the 
acquisition  system  must  be  able  to  respond.  In  some  cases,  cost  growth  results  from  prudent 
changes  in  quantity  or  capability  of  acquired  systems.  Our  ultimate  measure  of  performance  is 
providing  effective  systems  to  the  warfighter  that  are  suitable  for  fielding,  at  costs  that  are 
affordable,  while  ensuring  taxpayers'  money  is  spent  as  productively  as  possible.  Only  through 
rigorous  analysis  and  clear  reporting  will  we  be  able  to  separate  and  account  for  acceptable  and 
unacceptable  types  of  cost  growth,  informing  our  discussions  within  DoD,  with  Congress,  our 
Allies,  and  the  American  public. 


The  Honorable  Frank  Kendall 
Under  Secretary  of  Defense 
Acquisition,  Technology  and  Logistics 
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1.  Background 

Our  acquisition  system— its  institutions,  offices,  laboratories,  workforce,  managers,  executives, 
and  industrial  partners— conducts  research,  provides  a  wide  range  of  services,  develops  and 
produces  new  goods,  and  sustains  their  capabilities  for  warfighters  and  other  operators.  The 
performance  of  that  system  should  be  measured  relative  to  its  outputs  and  outcomes  of 
interest.  Identifying  internal  policies,  processes,  workforce,  and  management  capabilities  that 
bear  positively  or  negatively  on  those  measures  requires  data  and  analysis  to  avoid  speculative 
or  cyclical  policy  choices  based  on  current  conventional  wisdom  or  untested  hypotheses. 

Tracking  and  Disseminating  Acquisition  Performance  and  Correlates 

Measuring  performance  is  a  key  tool  for  improving  performance.  Fundamental  to  improving 
acquisition  is  to  understand,  in  sufficient  detail,  how  well  we  actually  are  performing:  whether 
we  are  getting  any  better  or  worse,  and,  most  importantly,  understanding  why  any  discernible 
trends  exist.  Without  baselines  and  trends,  it  is  impossible  to  know  where  to  focus  our  efforts, 
what  initiatives  should  be  considered  (or  abandoned),  and  whether  those  initiatives  are 
producing  the  desired  results. 

Determining  whether  we  have  improved  our  performance  is  a  challenging  task  because  many 
cost,  schedule,  and  performance  measures  are  lagging  indicators  that  require  years  to  reveal 
identifiable  trends.  Thus,  the  Department  must  continue  to  monitor  performance  closely  and 
seek  (whenever  possible)  more  responsive  or  even  leading  indicators  of  performance. 

At  the  direction  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(USD[AT&Lj),  the  Department  is  establishing  an  expanded  process  for  measuring  and  publishing 
the  performance  of  the  defense  acquisition  system,  including  the  performance  of  both 
government  and  industry  institutions  under  the  tenures  of  primary  decision  makers.  This 
reinforces  accountability  and  responsibility  throughout  the  acquisition  system.  The  expanded 
process  involves  a  three-pronged  approach  of  ensuring  objective  data  collection,  rigorous 
analysis,  and  unbiased  reporting  of  results.  Data  collection  is  a  strategic  project  well  under  way. 
Rigorous  analysis  is  supported  by  a  new  analysis  cell  in  AT&L  as  well  as  the  continued  use  of 
rigorous  and  objective  internal  and  external  analytic  entities.  Finally,  this  annual  report  is  a  key 
dissemination  vehicle  of  our  findings  and  begins  a  continuing  process  to  improve  our 
performance  through  data  acquisition  and  analysis.  Subsequent  reports  will  extend  and  expand 
these  findings  as  we  continue  to  improve  our  ability  to  objectively  measure  and  evaluate 
performance. 
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This  document  primarily  reports  on  the  results  of  our  data-driven  analysis.  Interpretation  of 
performance  and  the  implication  for  policies  are  left  largely  to  the  reader.  Some  aid  is  given 
after  the  tables  and  figures  to  help  the  reader  understand  the  analysis  and  to  highlight  key 
findings.  Unless  the  analysis  provides  clear  links  to  causes  and  policies,  the  document  merely 
reports  the  facts  to  inform  debates  elsewhere. 

Focus  of  the  Report 

The  defense  acquisition  system  acquires  goods  and  services  to  support  our  military  forces  both 
now  and  in  the  future,  while  fulfilling  our  responsibility  to  be  efficient  and  to  avoid  waste, 
fraud,  and  abuse  of  taxpayer  dollars.  The  DoD's  FY2014  base  budget  request  for  acquisition 
funding  totals  $166.8  billion,  of  which  $99.3  billion  is  for  Procurement,  and  $67.5  billion  is  for 
Research,  Development,  Test  and  Evaluation  (RDT&E)  programs.  Of  this  amount,  approximately 
40  percent  ($69.4  billion)  is  for  Major  Defense  Acquisition  Programs  (MDAPs),  which  provide 
the  bulk  of  the  readily  available  data  for  analysis  in  this  year's  report. 

While  using  MDAP  program-  and  contract-level  data,  we  focus  on  how  our  institutions 
performed  as  evidenced  by  the  collective  outcomes  of  these  MDAPs  along  various  dimensions 
and  acquisition  approaches  rather  than  on  the  individual  performance  of  these  programs. 

This  report  provides  improved  insights  into  key  aspects  of  total  life-cycle  cost  performance  to 
inform  policy  decisions  about  the  overall  acquisition  system  and  institutions.  There  are 
important  differences  in  how  programs  are  managed  depending  on  where  they  are  in  their  life 
cycles. 

Figure  1-1  depicts  the  entire  defense  acquisition  system  life  cycle2  and  the  portion  where  we 
currently  have  the  best  data  for  analysis,  namely  for  development  and  early  production.  While 
we  have  some  data  that  reflect  partially  on  the  performance  in  other  phases  (e.g.,  early 
research,  analysis  of  alternatives  [AoAs],  early  risk  reduction,  and  sustainment),  the  phases 
subsequent  to  early  production  are  reflected  at  best  by  early  estimates.  These  other  phases  will 
be  covered  in  subsequent  versions  of  this  report  as  we  improve  data  access,  quality,  and 
availability. 


See  https://dag.dau.mil/Pages/acqframework.aspx  for  a  description  of  the  life-cycle  phases  of  the  Defense 
Acquisition  Management  System  framework. 
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Figure  1-1.  The  Defense  Acquisition  System  Life  Cycle. 
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Judging  Benefits  Against  Cost 


This  report  seeks  objective  presentation  of  data  and 
analysis,  avoiding  claims  of  whether  the  outcomes  of 
technical  performance  and  schedule  are  worth  the  cost 
(generally)  and  cost  growth  (specifically).  It  is  important 
to  note,  however,  that  debates  about  value  are 
important  yet  must  be  informed  by  facts. 

The  excerpt  below  from  an  article  by  Col.  Mark  F. 
Cancian  (USMCR,  ret.)  illustrates  these  larger  external 
debates.  While  the  acquisition  of  the  original  six  Navy 
frigates  beginning  in  1794  is  a  case  where  value  can  be 
argued,  there  certainly  are  others  for  which  it  cannot. 
We  leave  it  to  the  reader  to  judge  the  cases  in  this 
report— informed  in  part  by  the  analysis  provided. 


Is  Cost  Growth  Always  Bad? 

Discussions  about  cost  growth  presume  that  it  is  always  bad  and  that  policy  makers 
should  take  drastic  actions  to  prevent  it.  A  cautionary  tale  from  the  early  days  of  the 
Republic  shows  that  the  situation  is  more  complicated  than  the  usual  morality  play 
about  shortsightedness  and  incompetence. 


3  This  1906  painting  by  G.T.  Margeson  depicts  the  USS  Constitution  sailing  past  the  dismasted  HMS  Guerriere  after 
action  on  August  19, 1812.  Official  U.S.  Navy  Photograph  428-N-1055208  (www.history.navy.mil). 
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In  1794,  the  young  United  States  authorized  the  construction  of  six  frigates  (United 
States,  President,  Congress,  Constitution,  Constellation,  and  Chesapeake).  Intended  to 
be  the  major  units  of  the  new  Navy,  the  ships  represented  the  aspirations  of  an 
ambitious  but  inexperienced  institution.  In  execution,  all  the  pathologies  of  today's 
weapon  systems  acquisition  were  evident.  Toll  [in  "The  Six  Frigates"]  (2006)  describes 
the  history  and  construction  of  these  ships. 

•  An  innovative  but  unconventional  design  was  criticized  as  "extravagant." 

•  A  multi-mission  requirement  for  irregular  warfare  (anti-piracy)  and  high- 
intensity  warfare  (against  major  powers  such  as  Great  Britain)  put  conflicting 
demands  on  the  design. 

•  Use  of  exotic  materials  delayed  construction  and  raised  costs.  (Key  hull 
components  required  live  oak,  which  had  to  be  imported  from  inaccessible 
coastal  areas  in  the  South.) 

•  A  divided  political  establishment  argued  over  the  need  and  cost. 

•  Contracts  were  spread  around  all  the  northeast  states  to  ensure  political 
support. 

•  Cost  growth  caused  schedule  slippage  and  program  instability. 

•  Congress,  alarmed  at  the  costs  and  delays,  conducted  inquiries  and  railed 
against  waste. 

But  the  story  did  not  end  there.  In  service,  the  ships  were  spectacular  successes.  Over 
the  course  of  their  careers,  they  fought  11  combat  actions,  winning  8  and  losing  3.  The 
exploits  of  the  Constitution  particularly  encouraged  the  young  nation.  These  successes 
were  achieved  while  badly  outnumbered  and  fighting  against  the  two  best  navies  in  the 
world— the  British  and  French.  How  was  this  possible?  The  advanced  design  that  caused 
so  many  problems  during  construction  also  gave  the  ships  a  decided  advantage  over 
other  ships  in  their  class.  They  could  defeat  any  ship  with  comparable  speed  and  outrun 
any  ship  that  was  more  powerful.  The  unexpectedly  high  cost  bought  capabilities  that 
proved  important  in  war. 

Substitute  for  frigates  the  M-l  tank,  F-15  fighter,  or  Ohio-class  submarine  and  the  story 
moves  forward  two  centuries.  All  of  these  programs  had  unexpectedly  high  costs,  but 
proved  world  class  in  operation.  The  existence  of  cost  growth  therefore  does  not 
necessarily  mean  that  the  acquisition  was  a  mistake. 


[from  Cancian,  2010] 


On  Measuring  Performance 


Institutional  performance  is  all  about  getting  value.  Value  to  the  Department  stems  from  the 
relative  benefits  (e.g.,  technical  performance)  of  the  goods  and  services  acquired  in  a 
responsive  time  (schedule)  compared  to  the  costs  to  the  taxpayer.  Hence,  measures  of  cost, 
schedule,  and  performance  serve  as  the  basis  for  measuring  the  effectiveness  of  the  acquisition 
system  in  converting  inputs  to  outputs  (see  Figure  1-2). 


Figure  1-2.  Output  Measures  for  the  Defense  Acquisition  System. 
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Integration  of  the  DoD  Decision  Support  Systems 

While  we  focus  on  the  defense  acquisition  system,  it  is  important  to  remember  that  our  acquisition 
system  interacts  with  other  systems  that  control  the  finances  and  requirements  for  the  items  being 
acquired.  Moreover,  all  three  of  these  systems  are  driven  by  broader  policies  and  strategies  created 
and  evolved  to  meet  the  missions  given  to  the  Department  by  the  country. 

The  Department  of  Defense  has  three  principal  decision-making  support  systems  for  meeting  its 
policies  and  strategy,  all  of  which  have  been  significantly  revised  over  the  past  few  years.  These 
systems  are  the  following: 

Planning.  Programming.  Budgeting  and  Execution  System  (PPBES):  The  Department's  process 
for  strategic  planning,  program  development,  and  resource  determination.  The  PPBES 
process  is  used  to  craft  plans  and  programs  that  satisfy  the  demands  of  the  National  Security 
Strategy  within  resource  constraints. 

Joint  Capabilities  Integration  and  Development  System  (JCIDS):  The  systematic  method 
established  by  the  Chairman  of  the  Joint  Chiefs  of  Staff  for  identifying,  assessing,  and 
prioritizing  gaps  in  joint  warfighting  capabilities  and  recommending  potential  solution 
approaches  to  resolve  these  gaps.  CJCS  Instruction  3170.01  and  the  JCIDS  Manual  describe 
the  policies  and  procedures  for  the  requirements  process. 

Defense  Acquisition  System:  The  management  process  by  which  the  Department  acquires 
weapon  systems,  automated  information  systems,  and  services.  Although  the  system  is 
based  on  centralized  policies  and  principles,  it  allows  for  decentralized  and  streamlined 
execution  of  acquisition  activities.  This  approach  provides  flexibility  and  encourages 
innovation,  while  maintaining  strict  emphasis  on  discipline  and  accountability.  A  significant 
revision  to  the  DoD  Instruction  5000.02,  the  governing  policy  on  operation  of  the  Defense 
Acquisition  System,  is  nearing  completion. 


Understanding  How  Internal  Functions  and  Processes  Affect  Performance 

The  acquisition  system  can  be  measured  at  two  fundamental  levels:  (1)  the  major  outputs  and 
outcomes  of  the  system,  and  (2)  the  key  functions,  responsible  entities,  and  institutions 
accountable  within  the  system  to  achieve  those  outputs  and  outcomes.  The  most  readily 
available  and  measurable  outcomes  assessed  throughout  the  report  are  cost  and  schedule 
growth,  but  some  readily  available  information  on  technical  performance  also  is  analyzed. 

Decomposing  the  acquisition  system  into  major  functional  responsibilities  enables  analysis  of 
how  elements  of  the  system  affect  the  ultimate  outcomes  of  interest.  Intermediate  outputs  and 
outcomes  of  key  institutional  functions  may  correlate  with  cost,  schedule,  and  performance 
outcomes,  but  others  may  be  too  small  or  difficult  to  discern  from  available  data.  Still,  a 
functional  decomposition  helps  to  understand  how  well  the  defense  acquisition  system 
performs,  based  on  management  principles  and  intermediary  outputs  and  outcomes.  As  this 
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work  moves  forward,  our  greatest  challenge  will  be  to  identify  the  relationships  between  and 
among  the  factors  the  Department  can  affect  (policies,  contract  terms,  incentives,  workforce 
skills,  etc.)  and  the  outcomes  we  are  trying  to  achieve.  This  report  is  a  step  in  that  process. 

Scope  of  Outcomes:  Programs  or  Their  Constituent  Contracts 

Our  analyses  often  examine  two  main  types  of  performance  data: 

•  Program-level  Data— describing  measurements  across  the  entire  program  (e.g., 
estimated  final  total  cost  growth  from  Milestone  B  baseline  for  all  units  to  be  procured), 
and 

•  Contract-level  Data— describing  measurements  on  one  of  the  many  contracts  that 
constitute  a  program  (e.g.,  the  total  cost  growth  from  original  negotiated  contract 
target  cost  for  an  early  lot  of  units  procured). 

Program-level  measures  show  how  well  the  acquisition  system  developed  the  ability  to  produce 
the  overall  program  against  original  estimated  baselines  despite  quantity  changes,  while 
providing  insight  into  whether  cost  growth  may  have  been  a  factor  in  quantity  changes. 

Contract-level  measures  provide  early  indicators  of  potential  program-level  issues  by  examining 
performance  when  the  Department  contracts  for  specific  work  from  industry.  Nearly  all  the 
actual  research,  development,  and  production  on  weapon  systems  are  performed  by  industry 
partners  through  contracts  with  the  Department.  Thus,  examining  performance  at  the  contract 
level  provides  detailed  and  potentially  useful  indicators  of  performance  that  eventually  will  be 
seen  at  the  more  aggregate  program  level. 

This  report  often  switches  between  these  types  of  data  as  we  examine  different  types  of 
institutions  (e.g.,  DoD-wide  to  military  departments  to  acquisition  commands)  and  different 
phases  of  acquisition  (e.g.,  development  or  early  production). 

While  contracts  are  the  key  execution  elements  of  a  program  (i.e.,  most  goods  and  even 
services  are  provided  by  contractors),  they  have  different  baselines  (e.g.,  contract  cost  targets) 
set  at  different  times  than  the  program's  Milestone  B  baseline.  Contract  performance  can  be 
earlier,  leading  indicators  of  the  larger  program's  performance  but  they  do  not  necessarily 
reflect  the  performance  revealed  in  program-level  measurements.  Thus,  it  is  critically 
important  to  recognize  what  type  of  data  is  being  discussed  at  each  point  in  the  report. 

Also,  care  must  be  taken  to  note  whether  cost  data  have  been  adjusted  for  inflation.  The 
available  program-level  budget  data  we  used  have  been  adjusted  for  inflation  (i.e.,  is  reported 
in  "base-year"  dollars),  but  the  contract-level  cost  growth  data  have  not  (i.e.,  are  only  reported 
in  "then-year"  dollars,  and  insufficient  temporal  information  was  available  for  us  to  adjust  the 
reported  figures  for  inflation).  Thus,  partly  because  of  inflation,  the  program-level  cost  growth 
figures  in  this  report  will  be  lower  than  those  for  contract-level  analyses. 
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Organization  of  the  Report 

The  chapters  that  follow  summarize  analyses  performed  on  MDAPs  and  their  contracts,  with 
the  goal  of  identifying  quantifiable  and  statistically  significant  correlations  between  cost  and 
schedule  growth,  and  other  measurable  factors.  These  analyses  indicate  areas  that  should  be 
the  focus  of  acquisition  leadership  to  identify  and  address  problems  to  improve  the  outcomes 
of  the  acquisition  process. 

Chapter  2  is  a  long  presentation  of  analytic  results.  It  begins  by  showing  cost  growth  on  major 
MDAP  contracts  with  contract  start  dates  plotted  against  government  acquisition  executive 
tenures  when  major  reviews  should  have  occurred.  It  then  shows  a  view  of  acquisition 
productivity  as  measured  by  program  cancellation  or  major  curtailment.  The  report  then 
discusses  performance  results  by  different  organizational  partitions  from  overall  DoD 
performance  through  military  departments,  commands,  and  finally  our  industry  partners.  When 
available,  analyses  of  the  causes  of  performance  differences  are  provided. 

Chapter  3  briefly  reviews  the  gaps  in  our  analysis,  important  policies  and  processes  we  want  to 
examine,  and  our  efforts  to  fill  data  gaps  and  further  our  analyses. 

Chapter  4  provides  some  comments  on  the  analyses.  However,  this  is  not  a  comprehensive 
review  of  defense  acquisition  performance.  Rather,  the  analytic  results  in  Chapter  2  constitute 
the  findings  of  the  report.  We  leave  it  to  the  reader  to  draw  his  or  her  own  conclusions  and 
observations  about  the  performance  of  the  defense  acquisition  system,  its  sufficiency,  and  the 
degree  of  progress  made  to  date.  As  you  will  see,  there  is  room  for  improvement  (as  is  always 
the  case),  and  Chapter  4  does  not  purport  to  excuse  weaknesses  or  claim  victory  for  strengths 
but  rather  raises  some  observations  for  larger  deliberations  outside  this  report. 


7 


Performance  of  the  Defense  Acquisition  System ,  2013 


(This  page  was  left  blank  intentionally) 


8 


Performance  of  the  Defense  Acquisition  System ,  2013 


2.  Analysis  of  Acquisition  Performance 

A  key  to  improving  acquisition  is  to  learn  from  our  successes  and  failures.  Without  looking  at 
results  of  past  actions,  we  have  no  feedback  to  know  whether  hypotheses  and  beliefs  pan  out 
in  the  complicated  world  of  defense  acquisition.  Objectively  examining  the  relative 
effectiveness  of  acquisition  components  and  institutions  while  attempting  to  distinguish  which 
factors  and  variables  affect  outcomes  not  only  allows  us  to  identify  successes  and  failures  but 
also  begins  to  lead  us  to  specific  lessons  we  can  try  to  replicate  and  control  points  we  can 
exploit. 

The  following  analyses  examine  key  outcome  dimensions  of  cost,  schedule,  and  technical 
performance  of  MDAPs  across  the  DoD,  Components,  acquisition  commands,  commodities,  and 
prime  contractor  spectrum,  often  measured  at  both  the  program  and  contract  level.  Combined, 
these  analyses  provide  insight  into  key  cause  and  effect  relationships,  focusing  attention  on 
problems  as  early  as  possible,  clarifying  misunderstandings,  and  informing  assessments  and 
learning. 

These  analyses,  however,  have  limitations  because  they  do  not  yet  fully  explain  the  underlying 
reasons  why  we  found  some  factors  to  be  significant  contributors  to  cost  or  schedule  growth. 
They  also  do  not  allow  us  to  judge  the  merits  of  decisions  that  led  to  program  or  contract 
changes  (e.g.,  whether  changes  were  due  to  poor  management  or  lack  of  discipline,  or  because 
of  disciplined  response  to  engineering  challenges  in  advancing  the  state-of-the-art  or  changing 
missions  and  threats).  Nevertheless,  these  measures  provide  insights  and  areas  for  further 
research  that  can  be  used  by  acquisition  professionals  as  a  part  of  management  and  oversight, 
as  well  as  offering  early  insights  of  potential  effects  that  may  be  realized  at  the  program  level. 

For  our  analyses  of  program  data,  recognize  that  the  MDAPs  examined  are  in  a  varying  state  of 
maturity— from  early  programs  that  may  or  may  not  develop  future  problems,  to  mature 
programs  adding  new  capabilities  to  existing  systems,  to  completed  programs. 

For  our  analyses  on  contract  data,  we  note  that  each  MDAP  may,  and  usually  does,  have  more 
than  one  major  contract  in  our  datasets.  These  contracts  may  be  for  development  or  early 
production.  No  attempt  was  made  to  ignore  outliers  (examining  extreme  successes  and  failures 
are  important  for  learning).  These  data  are  a  readily  available  way  to  dig  deeper  into  MDAP 
performance  below  the  common  program-level  measures.  We  often  report  medians  rather 
than  averages  (means),  given  the  skewed  distributions.4  In  such  cases,  medians  are  a  better 
measure  of  central  tendency  across  a  collection  of  contracts  while  averages  exaggerate  the 
effect  of  extreme  outliers. 


4  Part  of  the  skewing  in  the  distribution  of  cost  change  is  the  mathematical  boundary  of  cost  change  because  cost 
cannot  decrease  more  than  100  percent,  but  it  can  increase  more  than  100  percent. 
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Contract  Cost  Growth:  Possible  Relationship  to  Acquisition  Executive  Decisions 

Let  us  begin  by  presenting  cost  growth  on  major  MDAP  contract  over  the  last  20  years,  plotting 
contract  cost  growth  against  the  USD(AT&L)  in  office  at  each  contract  start  date.  These  data 
allow  us  to  introduce  the  magnitude  and  distributions  of  cost  growths  in  defense  acquisition 
and  the  challenges  of  trying  to  identify  causes  and  motivate  performance  ownership. 

Policy  and  execution  decisions  by  DoD  executives  should  bear  (in  part)  on  the  effectiveness  of 
the  overall  acquisition  system  during  their  tenures.  Such  decisions  include  changes  to  the 
defense  acquisition  system  policies  and  procedures  (e.g.,  through  changes  in  Departmental 
regulations);  approvals,  certifications,  and  exemptions  within  that  system;  institutional 
organization,  policies,  and  processes;  incentives;  personnel  selection,  training,  and  mentoring; 
guidance  and  execution  on  larger  programs,  including  acquisition  strategies  and  choices;  and 
myriad  other  effects.  More  specifically,  the  acquisition  executives  chair  the  boards  that  review 
programs  at  major  milestones,  guiding  both  program  directions  and  specific  approaches  to 
contracting.  Thus,  one  way  to  reveal  executive  effectiveness  is  to  measure  the  cost  and 
schedule  growth  on  major  contracts. 

Tracking  effectiveness  during  the  tenure  of  key  executives  may  help  baseline  the  effectiveness 
of  the  defense  acquisition  system  overall  and  could  offer  clues  for  further  research  into  broad 
policies  and  actions  leading  to  improved  practices.  Note  that  programs  started  in  the  most 
recent  tenures  (e.g.,  USDs  Frank  Kendall  and  Dr.  Ashton  Carter)  may  have  further  changes  in 
observed  effectiveness  due  to  the  relative  immaturity  of  these  efforts. 

Figures  2-1  and  2-2  show  approximately  20  years  of  contract  data  on  total  cost  growth  relative 
to  initial  contract  cost  targets  for  major  MDAP  development  and  early  production  contracts. 
Superimposed  on  the  scatter  charts  are  the  tenures  of  Defense  Acquisition  Executives  (DAEs)  in 
place  at  the  time  of  the  contract  start  date.  This  was  not  a  statistical  analysis  of  correlation 
between  DAE  and  contract  but  an  exploratory  examination  that  reveals  the  wide  variation  in 
cost  growth  of  major  contracts  and  possible  relationships  in  time.  An  assumption  was  made 
that  generally  a  contract  start  date  closely  follows  a  key  review  by  the  DAE  for  the  programs  to 
proceed  with  the  contract  (e.g.,  the  Pre-EMD  review  or  Milestone  B  decision),  although  there 
may  be  some  situations  where  this  was  not  the  case  and  other  factors  also  can  be  involved. 
Outliers  and  selected  contracts  for  well-known  programs  are  identified  by  program  name 
(including  some  duplicates  for  multiple  contracts  for  the  same  MDAP).  The  scatter  plot  reveals 
significant  variation  and  skewing  in  total  contract  cost  growth  measured  from  original  contract 
cost  target. 
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Figure  2-1.  DoD-Wide  Development  Contract  Total  Cost  Growth  and  USD(AT&L)  Tenures 
(1992-2011). 


Development  Contracts 


USD(AT&L) 


NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  MDA  to  approve  contract  award.  Army  programs  are  shown  in  green;  Navy  in  blue,  Air  Force  in 
red,  DoD-wide  in  purple  and  yellow.  Diamonds  on  the  charts  indicate  growths  mostly  attributable  to  the  start  of 
the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  Defense  Acquisition  Executive 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-2.  DoD-Wide  Early  Production  Contract  Total  Cost  Growth  and  USD(AT&L)  Tenures 
(1993-2011). 


Early  Production  Contracts 


USD(AT&L) 


Contract  Start  Date 


NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  MDA  to  approve  contract  award  Army  programs  are  shown  in  green;  Navy  in  blue,  Air  Force  in 
red,  DoD-wide  in  purple  and  yellow.  Diamonds  on  the  charts  indicate  growths  mostly  attributable  to  the  start  of 
the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  Defense  Acquisition  Executive 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Productivity:  Sunk  Costs  of  Canceled  Programs 

One  possible  productivity  measure  is  the  amount  of  RDT&E  funding  spent  on  major  programs 
that  were  canceled  before  producing  any  units  and  those  that  produced  many  fewer  units  than 
originally  planned.  Figure  2-3  shows  the  annual  percentage  of  RDT&E  funding  (less  earlier 
science  and  technology  expenditures)  by  each  Military  Department  on  major  weapon  system 
development  efforts  with  major  curtailment  or  no  operational  units  produced. 

Figure  2-3.  Major  Programs  Officially  Canceled  without  Producing  Any  or  Very  Few 
Operational  Units  (1995-2013). 


Source:  DoD  budget  data. 


Of  note,  the  Army  has  both  the  largest  number  of  canceled  programs  and  the  largest 
percentage  of  sunk  RDT&E  costs.  The  amount  of  funding  lost  was  relatively  constant  for  the 
Army  from  2004  through  2010,  coming  down  sharply  thereafter.  The  majority  of  the  Army's 
sunk  funding  problem  through  this  period  was  due  to  the  cancellation  of  the  Future  Combat 
System  (FCS);  however,  every  year  from  1996  to  2010,  the  Army  spent  more  than  $1  billion 
annually  on  programs  that  ultimately  were  canceled  (Decker  2011).  The  causes  of  these 
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program  cancellations  and  curtailments  were  not  examined,  but  an  overview  of  root-cause 
analyses  of  Nunn-McCurdy  cost  breaches  is  provided  later  in  this  chapter. 

Technical  Performance  of  MDAPs 

While  most  of  this  report  discusses  outcome  measures  of  cost  and  schedule,  this  section 
summarizes  some  readily  available  independent  assessments  of  technical  performance  of 
weapon  systems.  Future  reports  will  continue  to  expand  this  area. 

Mission  Effectiveness  and  Suitability  of  Acquired  Systems  by  Organization 

One  measure  of  technical  performance  of  acquisition  programs  is  how  they  rate,  as  a  group,  in 
operational  effectiveness  and  suitability  as  assessed  by  the  DoD  Director  of  Operational  Test 
and  Evaluation  (DOT&E).5 

As  a  high-level  indicator  of  the  technical  performance  of  major  programs,  this  report  uses 
"operational  effectiveness"  and  "operational  suitability."  Operational  effectiveness  is  defined  in 
the  JCIDS  Manual  as:  "Measure  of  the  overall  ability  of  a  system  to  accomplish  a  mission  when 
used  by  representative  personnel  in  the  environment  planned  or  expected  for  operational 
employment  of  the  system  considering  organization,  doctrine,  tactics,  supportability, 
survivability,  vulnerability,  and  threat."  Operational  suitability  is  a  composite  evaluation  that 
considers  a  system's  safety,  interoperability,  availability,  maintainability,  and  reliability.  It 
should  be  noted  that  operational  effectiveness  and  operational  suitability  are  not  measured 
solely  on  the  basis  of  system  technical  performance  parameters.  Rather,  measurements  are 
accomplished  through  an  evaluation  that  includes  the  system  under  test  and  all  interrelated 
systems  (its  planned  or  expected  environment  in  terms  of  weapons,  sensors,  command  and 
control,  and  platforms,  as  appropriate)  needed  to  accomplish  a  combat  mission. 

Robust  developmental  testing  occurs  throughout  the  earlier  phases  of  a  program's  life  cycle, 
intended  to  discover  and  correct  significant  issues  so  that,  by  the  time  operational  testing  is 
done,  discovery  of  major  technical  performance  issues  should  be  rare. 

The  following  table  and  graphs  summarize  DOT&E's  assessments  of  technical  performance  of 
weapon  systems  grouped  by  Military  Department,  and  how  these  have  trended  over  time  in  3- 
year  increments.  The  percentage  reported  represents  the  number  of  MDAPs  rated  Effective  or 
Suitable  divided  by  the  total  MDAPs  assessed  by  DOT&E.  These  results  were  taken  from 
DOT&E's  statutorily  required  Beyond  Low  Rate  Initial  Production  (BLRIP)  reports  done  prior  to 
any  decision  to  proceed  to  full-rate  production  of  an  MDAP.  Each  program  is  rated  (or  not)  as  a 
whole  as  Effective  and  Suitable. 


5  DOT&E  is  independent  statutorily  from  the  acquisition  organizations  and  is  responsible  for,  among  other  things, 
reporting  the  operational  test  results  for  all  MDAPs  to  the  Secretary  of  Defense,  USD(AT&L),  Service  Secretaries, 
and  Congress. 
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Figure  2-4.  Percent  of  DoD  MDAPs  Rated  as  Operationally  Effective  and  Suitable  (1984-2011). 


All  DoD  Programs 

^—Effective  —  •  Suitable 


(n=17)  (n=19)  (n=20)  (n=24)  (n=19)  (n=33)  (n=31) 

Source:  DOT&E  BLRIP  reports. 


Table  2-1.  Percent  of  MDAPs  by  Military  Department  Rated  as  Operationally  Effective  and 
Suitable  (1984-2011). 


Army 

Navy 

Air  Force 

(n=47) 

(n=73) 

(n=43) 

Effective 

94% 

88% 

84% 

Suitable 

77% 

81% 

51% 

Source:  DOT&E  BLRIP  reports. 
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Figure  2-5.  Percent  of  Programs  Rated  Effective  and  Suitable  by  Military  Department  (1984- 
2011). 


%  Effective  —  —Army  . Navy  Air  Force 


%  Suitable 


—  —Army  . Navy  - Air  Force 
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Mission  Effectiveness  and  Suitability  of  MDAPs  by  Commodity  Type 

Figure  2-6  depicts  aggregate  data  on  operational  test  results  since  DOT&E  was  established  in 
1984  broken  out  by  the  type  of  systems  (also  referred  to  as  a  commodity  within  the  DoD).  Note 
that  the  chart  includes  a  mix  of  programs  by  dollar  value  and  is  not  limited  solely  to  MDAP 
systems. 

Figure  2-6.  Program  Ratings  in  Operational  Testing  by  Commodity  Type  (1984-2011). 


Suitable 


Satellites 


NOTE:  Data  include  a  mix  of  Acquisition  Category  (ACAT)  l-lll  and  special  interest  programs.  The  size  of  the  circle 
indicates  the  number  of  programs  evaluated,  and  the  location  is  the  average  for  each  commodity  type. 

Source:  DOT&E  BLRIP  reports. 


When  examining  the  overall  ratings  by  commodity  type,  satellites  were  rated  the  best  at  100- 
percent  effective  and  suitable.  Ships  and  command,  control,  communications  and  intelligence 
(C3I)  systems,  on  the  other  hand,  had  the  lowest  ratings  at  about  70-  to  80-percent  rated 
initially  as  effective  and  60-  to  70-percent  initially  rated  as  suitable  based  on  operational 
testing.  Note  that  some  of  these  categories  have  very  small  datasets  and  thus  are  less  reliable 
to  generalize  (e.g.,  only  three  satellite  systems  are  in  the  dataset). 
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Cost  Growth:  Nunn-McCurdy  Program  Breaches 


Each  MDAP  is  required  by  law  to  submit  a  Selected  Acquisition  Report  (SAR)  to  the  Congress  45 
days  after  the  President's  annual  budget  submission  and  under  various  other  circumstances 
(see  10  U.S.C.  §2432).  The  SAR  reflects  what  is  included  in  the  President's  Budget  as  well  as  a 
comprehensive  summary  of  MDAP  cost,  schedule,  and  technical  performance  measures. 
Historical  SAR  data  serve  as  the  primary  sources  for  much  of  the  program-level  analysis  in  the 
report  due  to  their  relative  availability  and  comprehensiveness. 

Common  cost  measures  such  as  Program  Acquisition  Unit  Cost6  (PAUC),  which  includes  both 
RDT&E  and  procurement,  and  Average  Procurement  Unit  Cost7  (APUC),  which  includes  only 
procurement)  are  codified  in  statute.  Statute  also  requires  that  programs  exceeding  certain 
thresholds  (measured  by  PAUC  or  APUC  changes  relative  to  their  original  and  latest  program 
baselines)  must  go  through  a  rigorous  reexamination  and  certification  to  Congress  along  a 
variety  of  specified  criteria.  This  process  commonly  is  referred  to  as  the  "Nunn-McCurdy" 
process,  named  for  the  two  original  sponsors  of  the  legislation. 

Two  types  of  breaches  are  called  out  in  the  Nunn-McCurdy  process:  significant  and  critical.  A 
"significant"  breach  is  the  lower  threshold  and  is  intended  to  warn  Congress  that  a  program  is 
experiencing  high  unit  cost  growth.  A  "critical"  breach  signifies  the  cost  growth  is  even  higher, 
triggering  the  formal  reexamination  and  certification  process  mentioned  above.  The  criteria  for 
a  significant  breach  are  30  percent  cost  growth  in  APUC  or  PAUC  from  the  original  baseline  or 
15  percent  from  the  current  baseline  reported  in  the  previous  SAR.  A  critical  breach  occurs 
when  the  program  experiences  50  percent  cost  growth  from  the  original  baseline  or  25  percent 
from  the  current  baseline. 

Figure  2-7  shows  the  number  of  Nunn-McCurdy  breaches  since  1997,  including  each  individual 
breach  for  those  programs  that  have  breached  multiple  times  (e.g.,  a  significant  breach 
followed  by  a  critical  breach).  The  National  Defense  Authorization  Act  (NDAA)  for  FY2006  made 
changes  to  the  Nunn-McCurdy  statute,  adding  the  requirement  to  report  unit  cost  growth  from 
the  original  baseline;  this  additional  requirement  caused  the  large  spike  in  2005,  where  11 
programs  had  to  report  preexisting  significant  breaches.  There  have  been  86  total  breaches 
since  1997,  and  the  most  recent  years  reflect  an  apparent  improving  trend.  However,  it  is  too 
early  to  determine  if  this  trend  represents  a  systemic  improvement  in  performance. 


6  Section  2432(a)(1),  Title  10,  U.S.C.  defines  program  acquisition  unit  cost  as  "the  amount  equal  to  (A)  the  total  cost 
for  development  and  procurement  of,  and  system-specific  military  construction  for,  the  acquisition  program, 
divided  by  (B)  the  number  of  fully  configured  end  items  to  be  produced  for  the  acquisition  program." 

7  Section  2432(a)(2),  Title  10,  U.S.C.  defines  procurement  unit  cost  as  "the  amount  equal  to  (A)  the  total  of  all 
funds  programmed  to  be  available  for  obligation  for  procurement  for  the  program,  divided  by  (B)  the  number  of 
fully  configured  end  items  to  be  procured." 
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Figure  2-7.  Nunn-McCurdy  Breaches  (with  Multiple  Breach  Programs  1997-2012). 


1997  1998  1999  2000  2001  2002  2009  2004  2005  2006  2007  2008  2009  2010  2011  2012 

NOTE:  The  criteria  for  breaches  were  changed  in  NDAA  2006,  affecting  counts  starting  with  2005.  Breaches  are 
determined  using  "base-year"  dollars  (adjusted  for  inflation). 


Table  2-2  below  summarizes  a  different  analysis  of  Nunn-McCurdy  breaches  by  commodity.  In 
this  case,  we  do  not  "double  count"  programs  that  have  breached  multiple  times,  allowing  us  to 
compare  the  number  of  breached  programs  to  those  that  have  never  breached  and  the  total 
number  of  MDAPs  during  this  period. 
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Table  2-2.  Nunn-McCurdy  MDAP  Breaches  and  Rates  by  Commodity  Type  (Adjusted  for 
Multiple  Breach  Programs  1997-2011). 


Total # 
Programs 

# 

Breaches 

Breach 

Rate 

C3I 

33 

5 

15% 

Land  Vehicle 

9 

2 

22% 

Missiles 

22 

5 

23% 

Munitions 

12 

3 

25% 

Ships 

14 

4 

29% 

Satellites 

15 

5 

33% 

Other 

9 

3 

33% 

Fixed  Wing  Aircraft 

34 

12 

35% 

Submersible 

5 

2 

40% 

Helicopters 

13 

10 

77% 

Chem  Demil 

4 

4 

100% 

Total 

170 

55 

31% 

NOTE:  Breaches  are  determined  using  "base-year"  dollars  (adjusted  for  inflation). 


Thirty-one  percent  of  all  MDAPs  since  1997  have  had  either  a  significant  or  critical  breach.  This 
analysis  appears  to  show  that  all  commodities  are  susceptible  to  breaches,  with  helicopter  and 
the  chemical  weapons  demilitarization  (Chem  Demil)  programs  having  the  highest  breach  rates. 
The  Chem  Demil  programs  historically  had  unique  oversight  and  management  that  has  recently 
been  brought  more  in  line  with  practices  of  the  rest  of  the  Department;  the  cost,  schedule,  and 
technical  performance  metrics  were  not  mature,  and  were  subject  to  somewhat  unique  factors 
in  execution.  Helicopter  programs  show  a  clear  association  with  Nunn-McCurdy  breaches,  and 
this  is  an  area  that  requires  further  investigation. 

The  Nunn-McCurdy  process  provides  insights  but  only  comes  into  play  when  programs  already 
are  having  problems.  Also,  even  though  they  are  unit  metrics,  PAUC  and  APUC  are  sensitive  to 
quantity  changes,  which  can  mask  or  confuse  the  real  sources  of  cost  growth  or  other 
acquisition  problems.  Explaining  cost  growth  as  simply  a  result  of  changing  quantity,  therefore, 
can  be  complicated  and  misleading  without  careful  analysis.  The  SARs  do  contain  cost  variance 
discussions  that  provide  some  useful  explanations  of  changes  in  individual  programs,  but 
inconsistencies  in  the  variance  concepts  and  process  make  it  difficult  to  understand  root  causes 
of  cost  changes,  especially  across  multiple  programs  and  from  the  quantitative  data  in  the  cost 
variance  reports.  Recent  efforts  by  the  Department— especially  the  root-cause  analyses 
summarized  later  in  this  report— are  aimed  at  getting  beyond  mere  characterization  of 
"symptoms"  and  proximate  causes  to  gain  understanding  of  the  underlying  root  causes  and 
mechanisms  leading  to  cost  growth  on  programs. 
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Cost  Growth:  Development 

First,  let  us  examine  cost  growth  for  MDAPs  in  development.  MDAP  contract  data  provides  a 
valuable  baseline  of  historical  and  relatively  recent  cost  growth  behavior  to  begin  revealing  any 
near-term  and  long-term  trends  as  well  as  the  implications  of  common  practices.  The  long 
periods  examined  also  are  beneficial  because  they  provide  sufficient  data  for  statistical  analysis. 

Contract-Level  Development  Cost  Growth 

First,  let  us  revisit  cost  growth  at  the  contract  level  on  MDAPs  for  development.  Figure  2-8 
shows  total  cost  growth  since  about  FY1993  on  MDAP  contracts,  measured  from  the  originally 
negotiated  contract  target  costs.  (A  similar  chart  for  early  production  contracts  will  be 
discussed  in  the  next  section.)  This  figure  identifies  the  program  name  for  outliers  plus  selected 
programs  with  lower  total  cost  growth.  Note  that  contracts  shown  with  open  circles  indicate 
significant  cost  growth  well  beyond  the  contract  start  date,  which  could  indicate  factors  not 
attributable  to  the  initial  management  of  the  contract. 

Figure  2-8.  Contract  Level  Total  Cost  Growth  on  MDAP  Development  Contracts  (1992-2011). 


Development  Contracts 


NOTE:  Army  programs  are  shown  in  green,  Navy  are  blue,  Air  Force  are  red,  and  DoD-wide  are  in  purple  and 
yellow;  open  circles  indicate  that  there  was  significant  cost  growth  well  beyond  the  contract  start  date.  Source  cost 
data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 
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Program-Level  Development  Cost  Growth 

Generally,  RDT&E  costs  must  be  paid  regardless  of  how  many  units  are  produced.  In  that  sense, 
they  are  a  fixed  cost  for  the  Department  to  arrive  at  a  point  where  we  can  actually  procure  and 
field  a  capability.  Thus,  for  RDT&E,  cost  growth  ideally  should  be  tracked  in  total  rather  than  by 
unit  produced  to  avoid  confusing  the  effects  of  quantity  changes  with  RDT&E  cost  growth. 

The  following  figures  show  total  RDT&E  cost  growth  by  MDAP  portfolio  relative  to  the  original 
program  baseline  and  at  2-year  intervals.  These  different  views  are  useful  because  they  show 
how  the  portfolio  performed  from  inception  compared  to  how  it  performed  in  its  most  recent 
periods.  Differences  that  are  statistically  significant  are  indicated  with  asterisks. 

Examining  RDT&E  cost  growth  from  each  program's  original  baseline  estimate  is  important  to 
capture  the  overall  growth  since  inception;  however,  it  may  not  be  the  best  choice  to  gain 
insight  into  recent  cost  growth  management  because  MDAPs  can  have  very  long  lives.  When 
we  analyze  a  program  from  inception,  we  are  forced  to  carry  all  cost  growth  indefinitely. 
Programs  that  are  currently  executing  well  and  that  had  a  one-time  cost  increase  in  the  distant 
past  can  appear  to  be  long-term  poor  performers.  Therefore,  it  is  important  that  we  look  at 
both  types  of  data. 

Notably,  the  data  show  considerable  (and  sometimes  seemingly  conflicting)  differences 
between  the  medians  and  the  arithmetic  means.  This  is  because  the  data  are  highly  skewed, 
and  a  single  but  very  large  outlier  can  have  a  large  effect  on  the  mean  while  not  affecting  the 
median.8  Thus,  we  show  the  medians  to  provide  a  better  sense  of  the  central  tendency  of  the 
population,  and  we  provide  the  means  for  completeness  and  to  show  the  effects  of  these 
outliers.  Also,  these  values  are  not  weighted  by  program  dollar  value,  so  they  reflect  program 
effectiveness  generally  regardless  of  size. 

For  each  analysis,  we  first  show  the  main  portion  of  the  cost  growth  distribution  between  -10 
percent  and  100  percent  growth,  followed  by  a  separate  figure  showing  all  outliers  (especially 
those  with  growth  greater  than  100  percent).  The  "Box-and-whisker"  charts  show  the  20 
percentile,  median,  and  80  percentile  as  the  "box"  with  the  minimum  and  maximum 
constituting  the  "whiskers."  Gray-shaded  columns  in  the  table  beneath  each  chart  were  periods 
with  very  low  sample  counts  because  full  SAR  reporting  was  not  made  in  those  years  due  to 
new  Presidential  administrations.  The  "x"  markers  above  the  box  mark  the  five  largest  overruns 
(although  outliers  above  100  percent  only  appear  on  the  outlier  the  graphs). 


8  Part  of  the  skewing  in  the  distribution  of  cost  change  is  the  mathematical  boundary  of  cost  change  because  cost 
cannot  decrease  more  than  100  percent  but  can  increase  more  than  100  percent. 
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Figure  2-9.  RDT&E  Program  Cumulative  Cost  Growth  over  Original  Milestone  B  Baseline 
(2001-2011). 


Cost  Growth  (%) 


RDT&E  Program  Cumulative  Cost  Growth  over  Original  MS  B  Baselines 


2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011 

(n=78)  (n=74)  (n=76)  (n=83)  (n=81)  (n=87)  (n=90)  (n=13)  (n=94)  (n=97)  (n=90) 


Median 

**5% 

*  8% 

*  9% 

*  10% 

15% 

11% 

9% 

0% 

14% 

18% 

16% 

Mean 

86% 

51% 

54% 

58% 

65% 

81% 

80% 

75% 

82% 

80% 

84% 

*  Statistically  lower  than  2010 
**  Statistically  lower  than  2005-2011 


NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 


Figure  2-9  shows  that  RDT&E  total  cost  growth  has  been  statistically  flat  since  2005.  Growth  in 
2001  was  lower  than  in  2002-2004,  and  it  increased  after  that.  Thus,  the  median  growths  for 
2010  and  2011  (18  percent  and  16  percent,  respectively)  are  not  significantly  higher  than  even 
the  9  percent  in  2006.  You  can  see  that  visually  by  comparing  the  boxes.  This  emphasizes  the 
importance  of  statistically  examining  the  population  rather  than  just  considering  the  median  or 
mean. 
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Figure  2-10.  RDT&E  Program  Cumulative  Cost  Growth  Over  Original  Milestone  B  Baseline: 
Outliers  (2001-2011). 
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Largest  outlier 

GMLRS 
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C-130J 

C-130J 

2nd  Largest 

MH-60S  Knighthawk 

MH-6QS  Knighthawk 

3rd  Largest 

Global  Hawk 

GMLRS 
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4th  Largest 

CVN-77 

SBIRS 
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Global 

Hawk 
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Global  Hawk 

Global  Hawk 

5  th  Largest 

SBIRS 
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H-l 

Upgrades 

SBIRS 

High 

UH-6QM 

SBIRS  High 

SBIRS  High 

NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 


Note  that  the  maximum  cost-growth  percentages  are  very  high  due  to  a  small  number  of 
outliers  and  are  not  statistically  representative  of  the  overall  MDAP  portfolio.  These  extreme 
growths  are  not  due  to  measurement  error  so  were  not  excluded  from  the  analysis.  Still,  they 
do  skew  the  aggregate  data,  which  is  an  important  fact  for  how  to  measure  and  discuss  cost 
growth  across  a  program  population.  Interestingly,  similar  skewing  is  observed  in  various 
complex  commercial  projects  (see,  for  example,  Flyvbjerg  et  al.,  2002). 
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Understanding  why  a  program  may  exhibit  such  a  large  percentage  increase  in  RDT&E  cost 
requires  an  individual  examination  of  each  case.  For  example,  in  Figure  2-10,  the  C-130J  is  the 
highest  outlier  from  2002  through  2011.  The  program  originally  was  envisioned  as  a 
nondevelopmental  aircraft  acquisition  with  a  negligible  RDT&E  effort  planned.  Several  years 
into  the  program,  the  decision  was  made  to  install  the  Global  Air  Traffic  Management  system, 
adding  several  hundred  million  dollars  to  development  and  causing  the  total  development  cost 
growth  to  climb  upward  of  2,000  percent.  This  is  an  example  of  a  major  change  in  the  program 
rather  than  poor  planning  or  execution,  although  significant  program  changes  like  this  are  not 
necessarily  the  reason  for  all  extreme  cases  of  cost  growth. 


Figure  2-11.  RDT&E  Program  Cost  Growth  Over  2-Year  Spans  (2001-2011). 


NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 
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Comparing  growth  on  a  2-year  basis,  the  data  show  recent  statistically  significant  improvement 
in  the  2009-to-2011  period,  but  this  is  limited  evidence  of  an  improving  trend.  Further  analysis 
of  future  years  will  be  needed  to  see  if  this  trend  can  be  sustained. 


Figure  2-12.  RDT&E  Program  Cost  Growth  Over  2-Year  Spans:  Outliers  (2001-2011). 
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NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 
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Cost  Growth:  Early  Production 

Now  let  us  examine  cost  growth  for  MDAPs  in  early  production.  Again,  MDAP  contract  data 
provide  a  valuable  baseline  of  historical  and  relatively  recent  cost  growth  behavior  to  begin 
revealing  any  near-term  and  long-term  trends  as  well  as  the  implications  of  common  practices. 
The  long  periods  examined  also  are  beneficial  because  they  provide  sufficient  data  for 
statistical  analysis. 

Contract-Level  Early  Production  Cost  Growth 

First,  let  us  revisit  cost  growth  at  the  contract  level  on  MDAP— now  for  early  production.  Figure 
2-13  shows  total  cost  growth  since  about  FY1993  on  MDAP  contracts,  measured  from  the 
originally  negotiated  contract  target  costs.  This  figure  identifies  the  program  name  for  outliers 
plus  selected  programs  with  lower  total  cost  growth.  Note  that  contracts  shown  with  open 
circles  indicate  they  had  significant  cost  growth  well  beyond  their  start  dates,  which  could 
indicate  factors  not  attributable  to  the  initial  management  of  the  contracts. 


Figure  2-13.  DoD  Total  Cost  Growth  on  MDAP  Early  Production  Contracts  (1992-2011). 

Early  Production  Contracts 


NOTE:  Army  programs  are  shown  in  green,  Navy  are  blue,  Air  Force  are  red,  and  DoD-wide  are  in  purple  and 
yellow;  open  circles  indicate  that  there  was  significant  cost  growth  well  beyond  the  contract  start  date.  Source  cost 
data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Comparing  these  results  to  those  for  development  (e.g.,  Figure  2-8),  this  analysis  is  further 
evidence  that  we  are  much  better  at  executing  to  planned  costs  for  early  production  than  we 
are  for  development.  For  some  perspective,  historically,  since  1993,  total  contract  cost  growth 
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for  development  contracts  had  a  median  of  about  32  percent  (unadjusted  for  inflation).  The 
total  contract  cost  growth  for  early  production  had  about  a  9  percent  median  (unadjusted  for 
inflation). 

Program-Level  Early  Production  Cost  Growth  (Quantity-Adjusted) 

Now  at  the  program  level,  the  following  figures  summarize  the  unit  procurement  cost  growth 
across  the  MDAP  portfolio  from  the  original  MS  B  baseline  and  in  2-year  increments.  Again, 
box-and-whisker  charts  are  provided  for  both  the  main  portion  of  the  distribution  and  for 
outliers. 

These  program-level  data  are  for  unit  costs  and  (unlike  PAUC  and  APUC)  are  adjusted  for  any 
changes  in  procurement  quantity.  These  results  compare  recurring  procurement  unit  costs  at 
the  initially  estimated  quantities,  extrapolating  data  if  quantities  have  been  reduced.  This 
approach  provides  a  superior  way  of  comparing  what  the  units  would  have  cost  if  we  had  not 
changed  quantities  by,  essentially,  measuring  the  shift  in  the  cost-vs. -quantity  procurement 
cost  curve  from  planned  to  actual.  In  other  words,  we  measure  changes  in  procurement  cost  at 
the  currently  planned  quantity  to  be  purchased  (usually  lower  than  the  initial)  and  assume  that 
the  original  planned  quantity  still  was  being  purchased.  This  approach  allows  us  to  examine  on 
a  unit  basis  the  cost  of  the  capability  to  acquire  those  units  regardless  of  whether  we  increased 
or  decreased  quantity.  Of  course,  quantity  decreases  may  be  due  to  unit  cost  increases,  and  this 
approach  will  show  such  cost  increases  clearly. 
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Figure  2-14.  Program  Procurement  Cumulative  Unit  Cost  Growth  (Quantity  Adjusted)  Over 
Original  Milestone  B  Baseline  (2001-2011). 


Procurement  Program  Cumulative  Unit  Cost  Growth  over  Original  MS  B  Baseline 


Unit 

Cost  10o% 

Growth 

<at  90% 

original 

quantity) 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

-10% 


(quantity  adjusted) 


2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 
(n=14) (n=79) (n=76) (n=77) (n=83) (n=82) (n=88) (n=88) (n=ll) (n=91) (n=92) (n=84) (n=ll) 


Median 

1% 

6% 

8% 

7% 

3% 

4% 

6% 

9% 

S% 

8% 

8% 

5% 

0% 

Mean 

3% 

24% 

21% 

18% 

18% 

26% 

30% 

29% 

49% 

25% 

27% 

28% 

40% 

(differences  are  not  statistically  significant) 


NOTE:  The  analysis  only  includes  data  through  the  September  2012  Selective  Acquisition  Report  (i.e.,  the  annual 
2012  SAR  was  not  yet  available  at  the  time  of  this  analysis).  Source  budgetary  cost  data  were  reported  in  "base- 
year"  dollars  (adjusted  for  inflation). 
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Figure  2-15.  Program  Procurement  Cumulative  Unit  Cost  Growth  (Quantity  Adjusted)  Over 
Original  Milestone  B  Baseline:  Outliers  (2001-2011). 


NOTE:  The  analysis  only  includes  data  through  the  September  2012  SAR  (i.e.,  the  annual  2012  SAR  was  not  yet 
available  at  the  time  of  this  analysis).  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted 
for  inflation). 


Similar  to  the  prior  RDT&E  results,  cost  growths  are  highly  skewed  upward,  with  arithmetic 
means  higher  than  the  medians.  As  noted  above  for  the  contract-level  data,  the  overall 
magnitudes  of  the  cost  growths  are  not  nearly  as  large  as  those  for  RDT&E.  Also,  there  is 
considerable  variability  in  the  production  cost  growth  across  the  MDAP  portfolio. 
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Figure  2-16.  Program  Procurement  Cumulative  Unit  Cost  Growth  (Quantity  Adjusted)  Over  2- 
Year  Spans  (2001-2011). 


Procurement  Program  Unit  Cost  Growth  over  2-Year  Spans  (quantity  adjusted) 

Unit 


2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011 


Median 

e*4% 

12% 

2% 

i% 

i% 

i% 

2% 

4% 

0% 

2% 

*-i% 

Mean 

12%  1 

14% 

5% 

3% 

n% 

15% 

4% 

28% 

5% 

2% 

1% 

*  Statistically  lower  than  all  prior  two-year  periods 
**  Statistically  higher  than  all  periods  storting  2002  and  later 


NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 


Here  the  median  cost  growth  from  2009  to  2011  is  lower  statistically  than  all  prior  2-year 
periods,  again  highlighting  a  recent  improvement.  The  earliest  period  (1999-2001)  was 
statistically  higher  than  all  (unshaded)  periods. 
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Figure  2-17.  Procurement  Program  Cumulative  Unit  Cost  Growth  (Quantity  Adjusted)  Over  2- 
Year  Spans:  Outliers  (2001-2011). 


Procurement  Program  Unit  Cost  Growth  over  2-Year  Spans  (quantity  adjusted) 
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(n=65) 
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(n=62) 
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(n=7) 

(n=75) 
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(n=74) 

Largest  outlier 

CH-47F 

SBIRS 

High 

EELV 

NPOESS 

C-130 

AMP 

B-2  RMP 

C-130 

AMP 

2nd  Largest 

ATIRCM 

CMW5 

EELV 

CH-47F 

SBIRS 

High 

Land 

Warrior 

EFV 

AEHF 

SV  14 

JTRSHMS 

3rd  Largest 

GMLRS 

AW 

AEHF 

SV 1-4 

SBIRS 

High 

UH-60M 

SBIRS 

High 

NPOESS 

VTUAV 

Excalibur 

4th  Largest 

MH-60R 

Tactical 

Tomahawk 

AEHF 

SV  1-4 

FCS 

C-130 

AMP 

AEHF 

SV  14 

AB3A 

Reman 

FAB-T 

5th  Largest 

H-l 

Upgrades 

B-1BCMUP 

Computer 

UH-60M 

Land 

Warrior 

EFV 

LHA6 

H-l 

Upgrades 

WGS 

NOTE:  Source  budgetary  cost  data  were  reported  in  "base-year"  dollars  (adjusted  for  inflation). 
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Causes  of  Growth 


Let  us  now  review  existing  analysis  and  approaches  that  begin  to  indicate  major  causes  and 
components  of  cost  growth  on  MDAPs.  Independently  assessing  the  root  causes  behind 
significant  cost  growth  on  a  program  (or  even  a  single  contract)  requires  significant  effort 
beyond  the  data  readily  available  across  the  MDAP  portfolio.  Efforts  to  date  include  the  work  of 
the  Office  of  Performance  Assessments  and  Root  Cause  Analyses  (PARCA)  (especially  on  Nunn- 
McCurdy  breached  programs),  case  studies  of  individual  programs,  and  analyses  that  identify 
variables  that  correlate  statistically  with  cost  growth.  Cost  growths  often  are  examined  relative 
to  original  program  baselines  or  original  contract  cost  targets.  Conversely,  costs  could  be 
compared  to  those  for  legacy  weapon  systems  of  similar  types  to  obtain  an  absolute  measure 
of  the  cost  we  pay,  independent  of  acquisition  baselines. 

First,  we  will  review  trends  in  the  root-cause  analyses  performed  since  2010  by  the  PARCA  in 
AT&L  of  program  Nunn-McCurdy  breaches. 

Second,  we  will  review  findings  from  in-depth  analyses  of  all  contracts  on  six  selected  space 
system  programs.  This  analysis  breaks  contract  cost  growth  into  major  categories  of  work- 
content  growth  and  cost-over-target,  then  into  subcategories  that  help  show  what  constitutes 
these  major  categories. 

Third,  we  will  discuss  higher-level  analytic  results  that  use  readily  available  contract  data  to 
measure  work-content  growth  and  cost-over-target  across  all  major  MDAP  contracts  from 
1970-2011. 

Finally,  analyses  have  been  conducted  that  examine  a  range  of  contract  parameters  to  see 
which  parameters  correlate  statistically  with  cost  and  schedule  growth.  Some  of  these  variables 
begin  to  give  insights  into  the  causes  of  growth— at  least  at  a  high  level.  Combining  this  with 
analyses  that  compare  performance  between  institutions  can  start  to  explain  causes  of  cost 
growth  as  exhibited  by  these  institutions. 

PARCA  Root-Cause  Analyses  of  Program  Cost  Growth 

By  law,  AT&L  PARCA  must  perform  a  statutory  root-cause  analysis  for  all  "critical"  Nunn- 
McCurdy  breaches  as  well  as  discretionary  root-cause  analyses  requested  by  the  Secretary  of 
Defense  (see  10  U.S.C.,  Section  2438).  Table  2-3  summarizes  the  common  root  causes  in  the  18 
analyses  conducted  by  PARCA  over  the  last  3  years  (see  Bliss,  2012a;  Bliss,  2013). 
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Table  2-3.  PARCA  Root  Causes  Analyses  (Statutory  and  Discretionary;  2010-2012). 


10  of  18 
(56%) 


5  of  18 
(28%) 


Dominant _ 

Poor  management  performance 

•  Systems  engineering 

•  Contractual  incentives 

•  Risk  management 

•  Situational  awareness 

Baseline  cost  and  schedule  estimates 

•  Framing  assumptions 


4  of  18  Change  in  procurement  quantity 

(22%) _ 

_ Infrequent _ 

1  of  18  Immature  technology,  excessive  manufacturing,  or  integration  risk 

2  of  18  Unrealistic  performance  expectations 

1  of  18  Unanticipated  design,  engineering,  manufacturing  or  technology  issues 
None  Funding  inadequacy  or  instability 


Currently  Dominant  Root  Causes 

Poor  Management  Effectiveness.  The  broad  category  of  poor  management  effectiveness  was  a 
root  cause  in  just  over  half  of  the  cases.  Problem  areas  included: 

•  Poor  systems  engineering  to  translate  user  requirements  into  testable  specifications. 
This  includes  the  flow  down  of  requirements,  interface/environmental  management, 
and  management  of  holistic  performance  attributes  such  as  reliability  or  weight.  These 
largely  are  systems  engineering  functions. 

•  Ineffective  use  of  contractual  incentives.  This  includes  whether  the  acquisition  strategy 
selected  satisfies  the  conditions  necessary  for  its  success,  whether  it  is  consistent  with 
corporate  environment  (including  long-  and  short-term  objectives),  whether  it  is  aligned 
with  program  goals,  whether  there  are  perverse  effects,  and  whether  it  was  enforced. 

•  Poor  risk  management.  This  includes  the  identification,  quantification,  evaluation,  and 
mitigation  of  risks. 

•  Poor  situational  awareness.  Deficiencies  have  been  identified  in  program  office, 
contractor,  and  oversight  awareness,  and  the  timeliness  and  effectiveness  of  responses, 
related  to  the  cost,  schedule,  and  technical  performance  of  DoD  programs. 
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Baseline  Cost  and  Schedule  Estimates.  Baseline  cost  and  schedule  estimates  were  unrealistic 
in  just  over  one-fourth  of  the  cases.  The  primary  underlying  reason  was  invalid  framing 
assumptions  (Arena  et  al.,  2012;  Bliss,  2012b).  Framing  assumptions  are  any  explicit  or  implicit 
assumptions  central  in  shaping  cost,  schedule,  and/or  technical  performance  expectations.  A 
prototypical  example  of  a  framing  assumption  was  the  original  space  shuttle  processing 
concept  of  minimal  preparation  of  the  orbiter  between  launches  whereas  the  actual  processing 
involved  extensive  facilities  and  refurbishment  (e.g.,  individual  testing  of  each  heat  tile).  Below 
are  illustrative  examples  of  framing  assumptions  that  may  be  made  on  defense  systems: 

•  The  design  is  very  similar  to  the  prototype  or  demonstration  design. 

•  Modular  construction  will  result  in  significant  cost  savings. 

•  Arbitrating  joint  requirements  will  be  straightforward. 

•  The  satellite  bus  will  have  substantial  commercial  market  for  the  duration  of  program. 

A  recurring  problem  identified  in  the  PARCA  root-cause  analyses  of  Nunn-McCurdy  breaches  is 
a  lack  of  consideration  and  monitoring  of  program  framing  assumptions.  AT&L  is  working  with 
the  acquisition  community  to  establish  practices  that  explicitly  identify,  consider,  and  monitor 
key  framing  assumptions  upon  which  strategies  and  estimates  are  based.  This  effort  should 
enable  earlier  detection  and  adjustment  for  problems  that  lead  to  poor  cost,  schedule,  and 
technical  performance  of  MDAPs. 

Quantity  Changes.  Quantity  changes  for  reasons  outside  the  acquisition  community's  control 
caused  breaches  on  only  about  one-fifth  of  the  cases. 

Currently  Infrequent  Root  Causes 

Although  often  cited  as  common  acquisition  problems,  the  following  were  each  found  in  only 
one  case  each  (to  date): 

•  Immature  technology;  excessive  manufacturing  risk;  or  excessive  integration  risk. 

•  Unrealistic  performance  expectations. 

•  Unanticipated  design,  engineering,  manufacturing  or  technology  issues. 

Funding  inadequacy  or  instability  never  caused  a  breach  in  the  set  of  18  programs  examined  to 
date,  which  is  contrary  to  conventional  wisdom,  but  warrants  additional  analysis. 

These  results  provide  objective  insights  into  the  major  causes  of  cost  growth,  but  they  are  only 
performed  after  programs  have  problems.  Still,  some  lessons  learned  (e.g.,  the  framing 
assumption  process  discussed  above)  have  already  been  implemented,  but  more  work  is 
needed  on  the  underlying  issues  and  on  other  root  causes. 
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In-Depth  Case  Studies  of  Contract  Cost  Growth  on  Selected  Space  Programs 

Table  2-4  summarizes  the  results  of  a  case  study  of  six  space-related  MDAPs.  This  in-depth 
analysis  examined  all  contract  actions  on  the  six  programs,  using  expert  judgment  to  assign  cost 
growth  to  the  categories  identified. 


Table  2-4.  Comparing  Manual  Analysis  with  Contract  Cost  Growth  Analysis. 


Share  of  Total  Cost  Growth 

Six  Space  Programs 

All  Space 

All  DoD 

All  DoD 

Development 

Development 

Development 

Contracts 

Contracts 

Contracts 

Analysis: 

In- 

depth 

High-Level 

High-Level 

(1970-2011) 

High-Level 

(1970-2011) 

High-Level 

(1992-2011) 

Cost  Categories 

(37  contracts) 

(433  contracts) 

(176  contracts) 

Work  content  changes 

* 

0.25 

0.27 

0.40 

0.55 

0.69 

New  sub-tier  requirements 

0.08 

Additional  testing 

0.14 

Requirements  clarification 

0.04 

Requirements  descope 

Key  Performance 

-0.02 

Parameter  (KPP)9 
changes 

0.00 

Cost-over-target 

0.75 

0.73 

0.60 

0.45 

0.31 

Technology  development 

0.18 

Delays/ rework 

0.13 

Engineering  studies 

0.02 

Additional  testing 

0.02 

Design  changes 

0.01 

Integration 

0.23 

Payload 

0.08 

Vehicle 

0.07 

Command/Control 

0.07 

Contractor  execution 

0.34 

Rework 

0.14 

Design  flaws 

0.18 

Obsolescence 

0.01 

Supplier  delays 

0.01 

*Accounts  for  rounding  error  of  breakdown  categories  below. 

NOTE:  Analysis  of  the  six  space  programs  used  data  through  February  2011.  Analysis  of  high-level  data  used 
medians  to  determine  the  share  of  work-content-change  and  cost-over-target,  whereas  the  six  in-depth  case 
studies  were  based  on  aggregating  total  dollars. 
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While  these  studies  partition  costs  in  detail  to  lower-level  cost  categories,  they  are  very  time 
consuming  and  costly.  This  analysis  involved  deep  review  of  all  program  contracts  and 
consultation  with  program  and  contract  personnel. 

Also,  the  PARCA  root-cause  analyses  mentioned  earlier  have  the  benefit  of  avoiding  attribution 
to  proximate  causes  and  focus  on  the  largest  root  causes.  Their  focus  and  time  constraints, 
however,  do  not  allow  for  the  kind  of  complete  contract  change  allocation  of  small  portions  of 
cost  growth  in  the  way  other  case  studies  can. 

Primary  Elements  of  Contract  Cost  Growth 

In  addition  to  summarizing  the  case  studies.  Table  2-4  summarizes  analysis  of  high-level 
contract  and  PM  cost-estimate  data  readily  available  on  major  MDAP  contracts. 

The  comparison  in  the  table  shows  that  the  analysis  of  high-level  DoD  data  matches  well  at  the 
total  levels  with  each  of  the  six  in-depth  space  program  case  studies.  Thus,  these  six  space 
programs  case  studies  provide  valuable  insight  into  what  constitutes  work  content  changes  and 
costs-over-target.  They  also  provide  some  evidence  to  support  the  anecdotal  assertions  that 
Key  Performance  Parameter  (KPP)9  changes  are  very  infrequent. 

Note,  however,  that  the  split  of  total  cost  growth  between  work-content  growth  and  cost-over- 
target  in  these  six  case  studies  is  different  than  those  found  historically  going  back  to  1970  and 
very  different  from  the  median  behaviors  across  all  types  of  DoD  contracts.  These  six  programs 
had  about  a  quarter  of  their  cost  growth  from  work  content  changes  and  three-quarters  from 
cost  growth  over  the  contract  target  cost.  All  space  system  contracts  since  1970  showed  a  split 
of  about  40  and  60  percent,  respectively,  while  across  all  DoD  contracts  we  found  work  content 
to  be  the  larger  contributor  (55  percent  since  1970  rising  to  69  percent  in  the  last  2  decades). 
Thus,  these  six  space  programs  are  not  exemplars  of  the  split  of  cost  growth  between  work 
content  growth  and  cost-over  target  DoD-wide,  showing  yet  again  the  wide  range  of  outliers  in 
our  dataset. 

Approach.  Figure  2-18  illustrates  how  the  two  components  of  total  cost  growth— work  content 
growth  and  cost-over-target— relate  to  the  contract  cost  target  and  the  program  manager's 
estimate  at  completion  (PM-EAC).  All  the  contract  cost  data  used  in  our  analyses  are  reported 
in  "then-year"  dollars  and  are  thus  not  adjusted  for  inflation.  This  is  in  contract  to  the  program- 
level  budget  data  for  which  "base-year"  (inflation-adjusted)  data  are  available  and  used  in  our 
analyses. 

Note  that  the  target  cost  and  PM's  EAC  can  be  above  or  below  each  other  or  the  initial  target 
cost.  Thus,  work  content  growth  and  cost-over-target  can  be  positive,  zero,  or  negative.  The 


9  Key  Performance  Parameters  (KPPs)  are  key  system  attributes  or  characteristics  that  must  be  meet  in  order  for  a 
system  to  meet  its  operational  goals. 
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four  most  common  examples  of  these  seen  in  our  dataset  of  MDAP  development  contracts 
from  1970-2011  are  shown  on  the  right  side  of  Figure  2-18.  Note  also  that  work  content 
growth  and  cost-over-target  are  shown  as  positive  but  can  also  be  individually  or  both  negative 
(as  can  total  cost  growth). 


Figure  2-18.  Contract  Cost-Growth  Measures. 


Contract  Cost  Growth  Measures 

Final  PM’s  estimate  fno*  *°  sca^e) 

at  completion~\  """ 


Cost  over 
Target 

(+h 


Latest  contract 
target  cost  *y 


a* 


Work 

Content 

Growth 


Total 

Contract 

Cost 

Growth 


(+/-> 


Initial  contract 
target  cost  * 


*plus  estimated  target 
costs  for  authorized 
undefinitized  work 


Four  most  common  combinations 
on  development  contracts 

(out  of  433  contracts,  1970-2011) 


Occurs  257  times 


Total  Work  Cost 

cost  content  over 

growth  growth  target 


A 


t 


+ 


* 


Occurs  20  times 


Total 

Work 

Cost 

cost 

content 

over 

growth 

growth 

target 

t" 

+ 

Occurs  45  times 


Occurs  62  times 


Table  2-5  provides  the  equations  used  to  calculate  these  three  cost  growth  metrics.  The 
contract  budget  base  (CBB)  provides  the  target  cost  (plus  the  estimated  cost  of  any  authorized 
unpriced  work)  at  a  given  time  (t).  To  avoid  confusion  with  other  baselines  used  in  acquisition 
management,  we  simply  refer  to  these  as  the  contract  target  cost  instead  of  the  contract 
budget  base. 


Also,  Table  2-5  provides  a  description  and  the  possible  causes  of  changes  in  these  measures. 
The  case-study  analysis  in  Table  2-4  provides  analytic  support  for  this  decomposition.  As  the 
table  shows,  these  are  a  mixture  of  causes  that  could  stem  from  planning  poorly,  executing 
poorly,  or  consciously  choosing  to  alter  the  contract  to  respond  to  changes  in  requirements.  For 
example,  if  the  government  needs  to  change  deliverables  on  a  contract  to  reflect  normal 
engineering  design  changes,  the  negotiated  costs  for  those  changes  are  added  to  the  contract 
target.  If,  however,  a  contractor  finds  that  added  labor  is  needed  to  deliver  the  items 
sufficiently  specified  in  the  contract  (and  thus  reflected  in  the  existing,  negotiated  target  cost 
for  those  items),  then  those  costs  are  reflected  as  the  cost-over-target.  Program  management 
guidance  explicitly  precludes  adding  costs-over-target  into  the  target  cost,  and  our  analysis  of 
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target  cost  and  PM  EAC  profiles  indicates  such  ill-advised  practices,  if  occurring,  may  only  be 
happening  in  a  very  small  (single  digit)  percentage  of  MDAP  contracts  (see  last  bar  titled  "Poor 
contract  management"  in  Figure  2-28  and  Figure  2-43). 


Table  2-5.  Contract  Cost  Measure  Definitions  and  Example  Causes. 


Contract 

cost 

measures 

Definitions 

Description 

Possible  Causes 

(unordered) 

Total  cost 
growth 
(%) 

PM  EAC(t )  —  Target^ 0) 

Total  cost  growth  from 
original  contract  cost 
target  (plus  any  original 
authorized  unpriced 
work).  It  is  the  sum  of 
work  content  growth 
plus  cost-over-target 
growth. 

Any  additional  cost,  planned 
or  unplanned,  that  increases 
the  current  contract  cost 
from  the  original  cost. 

Target^ 0) 

Work 

content 

growth 

(%) 

Target(t)  —  Target^  0) 
Target(0) 

Portion  of  total  cost 
growth  due  to 
negotiated  contract 
work  content  changes 
as  reflected  by 
increases  in  the 
contract  target  cost. 

•  Planned  additional  work 
options  exercised 

•  System  specification 
changes 

•  Unforeseen  engineering 
changes  needed  to  meet 
requirements 

•  Quantity  changes 

•  Poor  understanding  of 
technical  maturity 

•  Requirement  changes10 

Cost- 

over- 

target (%) 

=  Total  cost  growth 

-  Work  content  growth 

PM  EAC(t )  —  Target(t ) 

Remainder  of  the  total 
cost  growth,  which  is 
due  to  costs  over 
current  target. 

•  Unrealized  assumptions 

•  External  factors  (e.g., 
higher  than  anticipated 
labor  or  material  costs) 

•  Poor  contractor 
performance 

Targeted) 

Table  2-6  summarizes  an  analysis  of  the  relative  contributions  of  the  two  elements  (work- 
content  growth  and  cost-over-target)  to  total  cost  growth  help  us  to  understand  the  magnitude 
of  their  relative  contributions.  A  limitation  of  the  available  data  is  that  while  we  can,  by 
inference,  determine  that  a  change  in  work  content  and  cost-over-target  did  happen,  the  data 


Some  analysis  to  date  (e.g.,  see  Table  2-4)  supports  the  assertions  that  changes  to  KPPs  are  infrequent,  but 
further  quantitative  analysis  is  warranted. 
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do  not  offer  visibility  into  what  specific  underlying  change  actually  happened,  nor  why  it 
happened  on  each  individual  case,  like  the  case  study  analysis  provides.  Detailed  case  studies 
would  be  necessary  to  examine  these  factors  in  more  detail. 


Table  2-6.  Frequency  of  Observed  Contract  Cost  Growth  Metrics  (1970-2011). 
433  development  contracts 


Number  of 

Number  of 

Number  of 

contracts 

contracts 

contracts 

Metric 

<0 

=0 

>0 

Work  content  growth 

39 

21 

373 

Cost  over  target 

61 

83 

289 

Total  cost  growth 

29 

11 

393 

440  early  production  contracts 


Number  of 

Number  of 

Number  of 

contracts 

contracts 

contracts 

Metric 

<0 

=0 

>0 

Work  content  growth 

66 

39 

335 

Cost  over  target 

146 

59 

235 

Total  cost  growth 

94 

11 

335 

NOTE:  These  figures  show  any  cost  growth  (negative  or  positive)  as  reported  to  the  precision  in  our  data. 


Factors  Tested  for  Correlation  with  Contract  Cost  and  Schedule  Growth 

Finally,  when  data  were  available  and  sufficient  for  statistical  analysis,  we  examined  what 
variables  or  factors  correlate  with  cost  and  schedule  growth  on  major  MDAP  contracts.  This 
analysis  allows  us  to  control  for  a  fairly  wide  range  of  possible  factors,  reporting  only  those  that 
correlate  with  the  growth  in  question.  It  also  allows  us  to  measure  how  much  of  the  variation  in 
the  data  can  be  explained  by  the  variables  we  examined  and  how  much  is  due  either  to  other 
variables  on  which  we  do  not  have  data  or  to  random  noise.  These  analyses  are  reported 
throughout  the  remainder  of  the  report. 

Table  2-7  lists  the  variables  that  are  generally  available  in  our  MDAP  contract  dataset  (i.e.,  we 
have  information  on  cost,  schedule,  system  complexity,  size,  definitization,  contract  type, 
system  commodity  type,  Military  Department,  and  prime  contractor  for  each  in  the  dataset). 
Nearly  all  analyses  examined  all  variables  except  prime  contractor,  which  were  only  examined 
for  analysis  related  to  industry  institutional  performance. 
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Table  2-7.  Variables  Examined  in  Statistical  Analyses. 


General  Category 

Variables 

Cost 

•  Total  cost  growth 

•  Work  content  growth 

•  Cost  growth  over  target 

•  Major  work  content  growth  timing  (early  or  later) 

•  Major  cost  growth  over  target 

Schedule 

•  Schedule  growth 

•  Cycle  time 

•  Percent  complete 

•  Start  date 

•  Trends 

System  complexity 

•  Complex  (binary) 

Size 

•  Size 

Definitization 

•  UCA  (binary) 

Contract  type 

•  Cost-plus 

-  CPAF 

-  CPFF 

-  CPIF 

•  Fixed-price 

-  FPIF 

-  FFP 

•  Hybrid 

System  commodity  type 

•  Aircraft  (including  rotary  wing  and  UAVs) 

•  Ships 

•  Space 

•  Engines 

•  Missiles 

•  Bombs 

•  Ground  tactical  vehicles 

•  Weapons 

•  Other 

Military  Department 

•  Army 

•  Navy 

•  Air  Force 

•  DoD  (joint) 

Prime  Contractor 

•  Contractor  (see  Table  2-24  for  groupings) 
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Trends  and  Correlates  of  Contract  Cost  Growth:  Development 
Contract  Cost  Growth  Trends:  Development 

Analysis  found  modest  but  statistically  significant  improvement  trends  in  contract  total  cost 
growth.  Comparing  the  decade  before  and  since  the  start  of  FY2002,  total  cost  growth  for 
development  contracts  has  been  10  percentage-points  lower  and  appears  to  be  trending 
favorably.  Comparing  contract  total  cost  growth  from  1992-2005  to  2005-2011,  the  reduction 
was  18  percentage  points. 

Although  all  these  contracts  had  completed  at  least  about  30  percent  of  original  contract  dollar 
amount  (55  percent  of  original  schedule),  some  caution  should  be  observed  because  many  of 
the  more  recent  contracts  have  not  completed  and  may  yet  end  with  cost  growths  higher  than 
currently  estimated. 

Figure  2-19  graphically  depicts  this  modest  downward  trend,  with  Figure  2-20  providing  an 
enlarged  view  of  the  dominant  portions  of  the  distribution.  The  trend  line  on  these  figures  is  a 
simple  linear  regression  through  all  the  cost  growth  points. 

Figure  2-19.  DoD  Total  Cost  Growth:  Development  Contracts  (1992-2011). 
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NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-20.  Enlarged  view  of  DoD  Total  Cost  Growth:  Development  Contracts  (1992-2011). 


NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Contract  Cost  Growth  Correlates:  Development 

The  following  analyses  identify  and  quantify  the  performance  differences  between  Military 
Departments,  weapon  system  commodity  types,  contracting  practices,  contract  size,  and 
schedule  effects.  These  analyses  examine  whether  any  of  these  variables  correlate  with  total 
cost  growth— or  whether  the  variation  in  cost  growth  is  confined  to  the  two  primary  cost 
growth  measures  independent  of  these  other  variables.  We  tested  all  these  variables  for 
statistical  correlation  with  total  cost  growth  first  for  all  DoD  development  contracts  from  1970 
to  2011,  then  conducted  separate  tests  for  contracts  to  develop  aircraft,  missiles,  munitions, 
space  systems,  and  ships. 

We  know  from  earlier  analyses  that  we  can  quickly  break  down  total  contract  cost  growth  into 
two  primary  measures:  work  content  growth  (increased  work  added  to  a  contract  and  reflected 
in  changes  to  the  contract's  target  cost)  and  cost-over-target  (as  reflected  by  the  PM  EAC;  see 
Figure  2-18).  We  also  know  generally  what  constitutes  these  two  measures  (even  if  more 
analysis  is  needed  on  each  contract  to  quantify  the  breakdown)  and  that  normally  (but  not 
always)  work  content  growth  is  the  larger  of  the  two  constituents  (see  Table  2-4). 

Because  work  content  is  a  constituent  of  total  cost  growth,  we  should  find  that  it  always 
correlates  positively  with  total  cost  growth.  The  following  regression  analyses  control  for  that 
effect  (and  thus  the  effect  of  the  complementary  cost  measure  of  cost-over-target),  allowing  us 
to  test  which  noncost  variables  (if  any)  correlate  with  total  cost  growth  while  controlling  for 


43 


Performance  of  the  Defense  Acquisition  System ,  2013 


cost  elements.  The  regressions  also  tell  us  how  large  the  contributions  are  from  each  variable 
when  the  other  variables  are  held  constant  (i.e.,  all  other  things  being  equal).  Thus,  we  have  a 
much  better  ability  to  test  for  and  measure  the  separate  institutional  and  process  performance 
differences  while  controlling  for  other  variable  effects. 

Table  2-8  summarizes  the  significant  variables  that  contribute  to  total  cost  growth  on  all  DoD 
development  contracts.  It  is  important  to  note  that  the  other  variables  listed  in  Table  2-7 
(contract  type,  other  commodity  types,  and  Military  Departments)  were  examined  and  found  to 
have  no  significant  contribution  statistically  to  total  cost  growth. 


Table  2-8.  Contributors  to  Total  Contract  Cost  Growth  on  DoD-Wide  Development  Contracts 
(1970-2011). 


Significant  variables 

Corresponding  effect  on 

Total  contract  cost  growth 

Work  content:  each  +1 
percentage-point 

+1.03  percentage-points 

Aircraft  contract 

+22  percentage-points 

UCA  at  any  time 

+7  percentage-points 

NOTE:  Work  content,  Aircraft,  and  UCAs  in  this  regression  explain  94  percent  of  the  observed  variation  in  total 
contract  cost  growth  (i.e.,  only  7  percent  of  the  variation  in  the  data  is  due  to  variables  we  do  not  have  data  for,  or 
is  random  noise).  Level  of  significance  is  5  percent  (only  5  percent  chance  we  cannot  reject  the  value  is  actually 
zero).  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 

As  expected,  the  effect  of  work  content  growth  is  statistically  significant,  but  we  also  found  that 
aircraft  contracts  also  exhibited  significantly  higher  total  cost  growth  historically  (generally  22 
percentage-points,  all  other  things  being  equal)  and  were  the  only  commodity  type  that  had  a 
measurable  effect.  Undefinitized  contract  actions  (UCAs)  also  had  a  smaller  but  measurable 
increase  on  total  cost  growth  (7  percentage  points)  historically. 

Correlates  by  Commodity  Type.  Table  2-9  summarizes  the  significant  variables  that  contribute 
to  total  cost  growth  on  development  contracts  for  five  different  system  commodity  types: 
aircraft,  missiles,  munitions,  space  systems,  and  ships. 
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Table  2-9.  Contributors  to  Total  Contract  Cost  Growth  on  Development  Contracts  by 
Commodity  Type  (1970-2011). 


Significant  variables 

Total  contract  cost 
growth  corresponding 

effect 

Aircraft 

(n=41) 

Work  content:  each  +1  percentage-point 

+1.08  percentage-points 

Missiles 

(n=157) 

Work  content:  each  +1  percentage-point 

+0.93  percentage-points 

Schedule  growth:  each  +1  percentage-point 

+0.19  percentage-points 

Army  contract 

+30  percentage-points 

Munitions 

(n=20) 

Work  content:  each  +1  percentage-point 

+0.79  percentage-points 

Space  systems 

(n=37) 

Work  content:  each  +1  percentage-point 

+1.13  percentage-points 

Ships 

(n=28) 

Work  content:  each  +1  percentage-point 

+0.86  percentage-points 

UCA  at  any  time 

+41  percentage-points 

NOTE:  The  variable  in  these  five  regressions  explain  96  percent,  93  percent,  69  percent,  91  percent,  and  39  percent 
of  the  observed  variation  in  total  contract  cost  growth,  respectively.  Other  variables  listed  in  Table  2-7  (schedule, 
size,  definitization,  contract  type,  other  system  commodity  types,  and  other  Military  Departments)  were  examined 

but  found  to  have  no 

significant  contribution  statistically  to  total  cost  growth. 

Source  cost  data  were  reported  in 

"then-year"  dollars  (unadjusted  for  inflation). 

As  expected,  the  effect  of  work  content  growth  is  statistically  significant  on  all  regressions,  but 
some  had  a  stronger  relationship  than  others.  We  also  found  other  variables  for  some 
commodity  types  that  also  exhibited  significantly  higher  total  cost  growth  historically. 
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For  missile  development  contracts,  we  found  a  small  but  statistically  significant  correlation 
between  schedule  growth  and  total  cost  growth  (each  percentage-point  increase  in  schedule 
generally  correlated  with  a  higher  total  cost  by  about  0.19  percentage  points).  Also,  total  cost 
growth  was  generally  30  points  higher  (all  other  things  equal)  on  Army  missile  development 
contracts.  This  was  the  only  Military  Department  with  a  statistically  significant  correlate  at  the 
commodity  level,  and  only  for  missile  development  contracts. 

For  ship  development  contracts,  we  found  a  statistically  significant  UCA  effect.  Thirty-nine 
percent  of  the  ship  development  contracts  had  a  UCA,  and  they  generally  add  41  percentage- 
points  to  total  cost  growth.  This  UCA  effect  was  larger  than  the  other  UCA  effects  identified  in 
this  report,  and  it  could  indicate  an  area  of  caution  and  attention  for  the  Navy. 

Temporal  Patterns  of  Cost  Growth:  Development 

We  examined  the  temporal  patterns  of  contract  cost  growth  on  176  large  development 
contracts  in  MDAPs  from  1993  to  2011  to  look  for  clues  to  problem  behaviors  and  possible 
causes  or  predictors  of  contract  cost  and  schedule  growth. 

Interestingly,  we  found  seven  general  types  of  cost  growth  patterns  when  looking  at  MDAP 
contract  data  (see  the  following  figures,  showing  an  illustrative  example  MDAP  for  each 
pattern).11  Each  pattern  is  suggestive  of  underlying  behaviors  in  how  the  contracts  were 
planned  or  managed:  well-managed,  premature  start,  premature  start  with  poor  estimate, 
work  added  later,  work  added  later  with  poor  estimate,  poor  estimate  only,  and  poor  contract 
management. 

Note  that  the  labels  for  each  pattern  are  a  general  interpretation  of  the  shape  of  the  pattern 
and  indicate  a  potential  cause  that  could  lead  to  the  pattern.  Further  analysis  would  be  needed 
on  each  program  to  determine  whether  the  label  applies  to  each  category.  For  example,  the 
"Poor  contract  management"  profile  indicates  a  suspicious  pattern  that  could  indicate  poor 
contract  management  wherein  cost  increases  may  be  rolled  into  the  target  or  where  the  work 
content  is  suspiciously  unstable.  Further  analysis  is  needed  to  determine  the  underlying  causes, 
but  this  kind  of  pattern  analysis  is  one  way  to  quickly  categorize  contract  behavior  and  focus 
attention  on  contracts  that  indicate  a  possible  concern. 

Also,  each  program  selected  to  be  shown  as  the  exemplar  was  not  selected  for  any  particular 
reason  from  the  others  that  could  have  been  shown.  Still,  it  is  informative  to  name  the  program 
from  which  the  contract  came.  For  example,  the  F-35  EMD  airframe  contract  example  shows 
that  the  cost  growth  for  JSF  development  is  coming  mostly  from  cost-over-target  growth  rather 
than  work  added  to  the  contract  by  the  government. 


Note  that  the  difference  between  work  content  growth  and  total  cost  growth  is,  by  definition,  cost-over-target. 
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Figure  2-21.  Example  Development  Contract  Cost-Growth  Profiles:  "Well  Managed." 
Well-Managed  (F/A-18E/F  EMD) 


Work  content  Total  cost 

growth  growth 


Figure  2-22.  Example  development  contract  cost-growth  profiles:  "Premature  Start." 


Premature  start  (DDG-1000:  First  two  lead-ship  awards) 
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Figure  2-23.  Example  Development  Contract  Cost-Growth  Profiles:  "Premature  Start  +  Poor 
Estimate." 


Figure  2-24.  Example  Development  Contract  Cost-Growth  Profiles:  "Work  Added  Later." 
Work  added  later  (C-17  improvement  started  2001) 


Work  content  Total  cost 

growth  growth 
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Figure  2-25.  Example  Development  Contract  Cost-Growth  Profiles:  "Work  Added  Later  +  Poor 
Estimate." 


Work  added  later;  poor  estimate  (JSOW  EMD) 


Growth  from 
initial  contract 
budget  base 


Work  content  ,  Total  cost 
growth  growth 


Figure  2-26.  Example  Development  Contract  Cost-Growth  Profiles:  "Poor  Estimate." 
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Figure  2-27.  Example  Development  Contract  Cost-Growth  Profiles:  "Apparently  Poor 
Management." 


Apparent  Poor  Contract  Management  (Stryker  EMD) 


Figure  2-28.  Summary  of  Development  Contract  Cost-Growth  Profiles. 


Well-managed  Premature  Premature  Major  work  Major  work  Poor  cost  Poor  contract 

contracts  start  start /Poor  added  later  added  later/  estimate  management 

cost  estimate  Poor  cost 

estimate 


NOTE:  n=176  total  development  contracts. 
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The  frequency  of  each  pattern  in  Figure  2-43  reveals  how  common  these  problems  appear 
historically  across  MDAPs.  Just  over  a  quarter  of  the  major  contracts  (50  out  of  176,  or  28 
percent)  were  well  managed  with  minimal  cost  growth  of  any  kind.  The  remaining  contracts 
had  some  type  of  large  cost  growth  in  one  of  the  six  categories. 

The  largest  cost-growth  patterns  by  count  were  the  addition  of  major  work  to  37  contracts  later 
in  contract  execution  (21  percent),  and  32  contracts  (18  percent )  solely  experiencing  cost-over- 
target  (interpreted  as  poor  cost  estimates).  Fourteen  contracts  (8  percent)  exhibited  cost- 
growth  patterns  that  may  indicate  poor  contract  management. 

Early  Contract  Work  Content  Stability:  Development 

For  those  patterns  discussed  above  where  work  content  was  stable  in  the  first  year,  we  found 
that  cost  and  schedule  growth  also  were  more  stable.  The  following  are  the  statistical  results  of 
examining  MDAP  development  contracts  from  June  1992  through  December  2011. 

•  Development  contracts  with  early  work  content  stability  exhibited  significantly  lower: 

■  Total  cost  growth  -61  percentage  points 

■  Work  content  growth  -82  percentage  points 

■  Schedule  growth  -31  percentage  points 

(all  at  the  median  and  all  other  things  equal;  costs  unadjusted  for  inflation). 

This  helps  confirm  the  view  that  a  well-understood  and  -defined  contract  at  the  outset  tends  to 
perform  better  in  the  long  term.  Note  that  we  found  no  statistically  significant  difference  in 
costs-over-target  on  development  contracts  between  stable  and  unstable  work  content  in  the 
first  year  of  the  contract.  Also  remember  that  it  can  take  some  time  for  problems  on  such 
contracts  to  be  revealed. 

For  this  analysis,  we  only  included  data  on  contracts  that  spent  about  a  third  of  their  original 
cost  target  (nominally  about  55  percent  of  their  original  schedule).  Analysts  generally  find  that 
the  cost  estimates  for  the  completed  project  become  reliable  at  this  point. 

Effect  of  Contract  Type  Contract  Cost  and  Schedule  Growth:  Development 

Analysis  of  historical  MDAP  contracts  revealed  no  statistical  correlation  between  the  use  of 
contract  type  (e.g.,  cost-plus  and  fixed-price  types)  and  lower  cost  or  schedule  growth.  In  other 
words,  the  type  of  contract  used  in  each  case  did  not  result  in  a  statistically  significant 
difference  in  cost  growth.  No  differences  were  found  in  contract  type  in  433  MDAP 
development  contracts  from  1970-2011.  (We  found  and  discuss  similar  results  later  in  the 
report  for  440  MDAP  early  production  contracts.)  This  analysis  is  depicted  in  the  following 
tables  and  charts. 
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Table  2-10.  Fixed-Price  and  Cost-Type  Contract  Effects  on  Total  Development  Contract  Cost 
Growth  (1970-2011). 


Significant  variables 

Corresponding  effect  on 

Total  contract  cost  growth 

Work  content:  each  +1  percentage  point 

+1.03  percentage  points 

Aircraft  contract 

+22  percentage  points 

UCA  at  any  time 

+7  percentage  points 

Insignificant  variables  (no  statistical 
significant  effects  on  total  contract  cost 
growth) 

I  Contract 

I 

•  type 

« 

% 

\ 


•  Cost-plus 

-  CPAF 

-  CPFF 

-  CPIF 

•  Fixed-price 

-  FPIF 

-  FFP 

•  Hybrid 


\ 


» 

/ 


Controlling  for  the  three  variables 
above ,  fixed-price  contracts  did  not 
exhibit  a  significantly  different  cost 
growth  than  cost-reimbursable 
contracts. 


"SRTps 


System 

commodity 

type 


•  Space 

•  Engines 

•  Missiles 

•  Bombs 

•  Ground  tactical  vehicles 

•  Weapons 

•  Other 


•  Army 

Military  •  Navy 

Department  •  Air  Force 

•  DoD  (joint) 

NOTE:  The  three  statistically  significant  variables  in  the  regression  (work  content,  aircraft,  and  UCAs)  explain  94 
percent  of  the  observed  variation  in  total  contract  cost  growth.  The  level  of  significance  is  5  percent  (i.e.,  there  is 
only  a  5  percent  chance  we  cannot  reject  the  value  is  actually  zero).  Source  cost  data  were  reported  in  "then-year" 
dollars  (unadjusted  for  inflation). 


Table  2-10  shows  the  three  variables  that  are  statistically  correlated  with  total  contract  cost 
growth  along  with  the  other  variables  we  tested.  Note  that  contract  type  did  not  correlate  with 
total  contract  cost  growth. 


52 


Performance  of  the  Defense  Acquisition  System ,  2013 


Figure  2-29.  Contract-Type  Effects  on  Development  Contract  Cost  Growth  (1970-2011). 
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Figure  2-30.  Contract-Type  Effects  on  Development  Contract  Schedule  Growth  (1970-2011). 
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Statistical  analysis  also  confirmed  that  there  is  no  statistically  significant  difference  in  Figure 
2-29  between  cost-reimbursable  and  fixed-price  contracts  with  respect  to  schedule  growth  on 
development  contracts. 


Effect  of  UCAs  on  Cost  Growth:  Development 

Undefinitized  contract  actions  (UCAs)  had  a  measurable  increase  on  total  contract  cost  growth 
and  also  on  cycle  time  in  development.  Using  a  binary  flag  for  whether  a  major  MDAP  contract 
had  a  UCA,  analyses  found  that  a  contract  with  a  UCA  would  be  expected  to  have  7  percentage 
points  higher  total  cost  growth  (all  other  things  being  equal)  as  measured  across  all  DoD 
contracts  from  1970-2011.  UCAs  effects  on  ship  development  contracts  were  much  larger:  41 
percentage  points  higher  total  cost  growth  (again,  all  other  things  being  equal)  as  measured 
across  all  DoD  ship  development  contracts  from  1970  to  2011.  UCAs  effects  were  not 
statistically  significant  on  aircraft,  missile,  munitions,  or  space  systems  development  contracts. 

As  we  will  see  shortly,  a  contract  with  a  UCA  generally  lasted  0.3  years  longer  (all  other  things 
being  equal),  as  measured  across  all  DoD  contracts  from  1970-2011.  Thus,  UCAs  increase 
development  cycle  time  by  increasing  schedule  growth. 
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Cycle  Times:  Development 

We  are  examining  whether  it  is  taking  longer  to  develop  new  systems  and  why.  Statistical 
analysis  of  the  length  of  development  contracts  for  433  major  development  MDAP  contracts 
between  1970  and  2011  is  summarized  in  the  following  figures  and  tables. 

First,  Figure  2-31  shows  the  individual  factors  from  an  analysis  of  multiple  variables  and  the  size 
of  their  independent  contributions.  The  individual  contributions  can  be  added  together  to  yield 
the  expected  cycle  time  of  a  contract  based  on  historical  data  and  trends.  In  other  words,  given 
a  certain  contract,  you  can  add  the  cycle  time  contributions  of  these  individual  factors  based  on 
whether  these  factors  are  present  in  the  contract  and  (in  the  case  of  the  two  cost-growth 
factors)  by  how  much. 

All  other  things  equal,  development  cycle  time  on  contracts  after  1980  took  an  average  of  0.9 
years  longer  than  contracts  before  1980— an  increase  of  about  one-sixth  over  the  base  of  5.2 
years. 

This  analysis  also  found  some  smaller  but  still  statistically  significant  correlations  between  cycle 
time  increases  and  cost  growth  on  MDAP  contracts.  Every  10  percentage-point  increase  in 
work-content  cost  growth  generally  added  0.066  years,  and  every  10  percentage-point  increase 
in  cost-over-target  generally  added  0.16  years. 

Also,  contracts  with  UCAs  were  about  0.3  years  longer  generally. 

Contracts  for  space  systems  were  an  additional  1.7  years  longer,  whereas  contracts  for  aircraft 
were  2.5  years  longer.  No  other  commodity  types  had  significantly  longer  cycle  times. 

Thus,  for  example,  a  contract  started  before  1980  with  a  UCA,  10  percentage  points  increase  in 
work  content,  90  percentage  points  increase  in  cost-over-target,  and  for  a  space  system  would 
statistically  be  expected  to  have  a  cycle  time  (contract  length)  of  8.7  years.12  Conversely,  a 
contract  started  after  1980  without  a  UCA,  with  120  percentage  points  increase  in  work 
content,  15  percentage  points  increase  in  cost-over-target,  and  which  was  not  for  a  space 
system  or  aircraft  statistically  would  be  expected  to  have  a  cycle  time  (contract  length)  of  7.1 
years.13 


12  5.2  +  0.3  +  0.066  +  (9*0.16)  +  1.7  =  8.7 

13  5.2  +  0  +  0.9  +  (12*0.066)  +  (1.5*0.16)  +  0  =  7.1 
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Figure  2-31.  Contributors  to  Development  Contract  Cycle  Time  (Length)  DoD-Wide  (1970- 

2011). 
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Every  +10  percentage-point  increase  in  work  content  cost  growth  generally  added  +0.066  years 
Every  +10  percentage-point  increase  in  cost-over-target  generall  added  +0.16  years 


NOTE:  This  analysis  used  the  length  of  433  MDAP  development  contracts  as  contract  cycle  time.  The  regression  of 
two  simultaneous  equations  examined  the  indirect  effect  of  schedule  growth  on  cycle  time.  Other  variables 
examined  (see  Table  2-10)  were  not  statistically  significant  contributors  to  cycle  time  and  are  not  shown  here.  The 
level  of  significance  for  each  variable  was  <=5%  (i.e.,  there  is  at  most  a  5  percent  chance  we  cannot  reject  that  the 
value  actually  is  zero). 
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Figure  2-32  shows  a  simpler  before/after  1980  analysis  of  cycle  time  that  did  not  control  for 
other  variables  but  shows  the  medians,  percentiles,  and  frequency  distributions  of  these 
development  contracts.  Here  the  median  difference  of  1  year  is  very  close  to  the  0.9  years  in 
the  first  analysis. 

Additional  sensitivity  analysis  on  the  before/after  date  for  different  commodity  types  and 
Military  Departments  yielded  some  additional  statistically  significant  results. 

•  Aircraft  development  cycle  times  also  increased  significantly  after  1980  (see  Figure 
2-33). 

•  Space  system  development  cycle  times  increased  at  the  1995  point  instead  (see  Figure 
2-34). 

•  "Other"  commodity  types  (including  include  electronics,  avionics,  and  communication 
systems)  decreased  significantly  after  1995  (see  Figure  2-35). 

All  other  differences  in  cycle  times  were  not  statistically  significant  (e.g.,  some  due  to 
insufficient  sample  size),  and  remaining  variation  in  the  data  is  due  to  variables  outside  our 
dataset,  or  random  noise. 

Additional  analysis  of  cycle  times  by  Military  Department  are  discusses  later  in  this  chapter. 
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Figure  2-32.  All  Development  Contract  Cycle  Time  Trends  (1970-2011). 

All  development  contract  cycle  times  increased  significantly  after  1980 
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NOTE:  The  bottom  "box  and  whisker"  chart  shows  the  middle  two  quartiles  in  the  box  with  the  median.  The 
"whiskers"  show  the  maximum  and  minimum  values  in  the  table  at  the  top  of  the  figure. 
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Figure  2-33.  Aircraft  Development  Contract  Cycle  Time  Trends  (1970-2011). 

Aircraft  development  contract  cycle  times  increased  significantly  after  1980 
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NOTE:  Aircraft  contracts  after  1980  included  B-2,  C-17,  A-12,  F-22,  V- 22,  F-35,  Global  Flawk,  and  Comanche,  among 
others.  The  bottom  "box  and  whisker"  chart  shows  the  middle  two  quartiles  in  the  box  with  the  median.  The 
"whiskers"  show  the  maximum  and  minimum  values  in  the  table  at  the  top  of  the  figure. 
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Figure  2-34.  Space  Development  Contract  Cycle  Time  Trends  (1970-2011). 

Space  development  contract  cycle  times  increased  significantly  after  1995 
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Figure  2-35.  "Other"  Commodity  Type  Development  Contract  Cycle  Time  Trends  (1970-2011). 

"Other"  commodity  type  development  contract  cycle  times  decreased  significantly  after  1995 
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NOTE:  "Other"  commodity  types  include  electronics,  avionics,  and  communication  systems.  The  bottom  "box  and 
whisker"  chart  shows  the  middle  two  quartiles  in  the  box  with  the  median.  The  "whiskers"  show  the  maximum  and 
minimum  values  in  the  table  at  the  top  of  the  figure. 
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Trends  and  Correlates  of  Contract  Cost  Growth:  Early  Production 
Contract  Cost  Growth  Correlates:  Early  Production 

As  we  did  for  development  contracts,  we  also  identified  and  quantified  early  production 
contracts  performance  differences  between  Military  Departments,  weapon  system  commodity 
types,  contracting  practices,  contract  size,  and  schedule  effects.  These  analyses  examine 
whether  any  of  these  variables  correlate  statistically  with  total  cost  growth— or  whether  the 
variation  in  cost  growth  is  confined  to  the  two  primary  cost  growth  measures  independent  of 
these  other  variables.  We  tested  all  these  variables  for  statistical  correlation  with  total  cost 
growth  first  for  all  DoD  development  contracts  from  1970  to  2011,  then  conducted  separate 
tests  for  early  production  contracts  for  aircraft,  missiles,  munitions,  space  systems,  and  ships. 

Table  2-11  summarizes  the  significant  variables  that  contribute  to  total  cost  growth  on  early 
production  contracts.  It  is  important  to  note  that  other  variables  listed  in  Table  2-7  (contract 
type,  other  commodity  types,  and  Military  Departments)  were  examined  and  found  to  have  no 
significant  contribution  statistically  to  total  cost  growth. 


Table  2-11.  Contributors  to  Total  Cost  Growth  on  DoD-Wide  Early  Production  Contracts 
(1970-2011). 


Significant  variables 

Corresponding  effect  on 
total  contract  cost  growth 

Work  content:  each  +1 
percentage-point 

+1.07  percentage-points 

Schedule  growth:  each  +1 
percentage-point 

+0.09  percentage-points 

Army  contract 

+12  percentage-points 

NOTE:  Work  content,  schedule  growth,  and  Army  contracts  in  this  regression  explain  92  percent  of  the  observed 
variation  in  total  contract  cost  growth  (i.e.,  only  8  percent  of  the  variation  in  the  data  is  due  to  variables  we  do  not 
have  data  for,  or  is  random  noise).  Level  of  significance  is  5  percent  (only  5  percent  chance  we  cannot  reject  the 
value  is  actually  zero).  Source  cost  data  were  reported  in  ''then-year"  dollars  (unadjusted  for  inflation). 


As  expected  and  discussed  earlier  on  development  contracts,  the  effect  of  work  content  growth 
dwarfed  all  others,  but  unlike  on  DoD-wide  development  contracts  now  there  is  a  small  but 
measurable  schedule  effect,  and  Army  contracts  now  have  a  statistically  significant  effect. 
Interestingly,  UCAs  did  not  correlate  with  total  cost  growth  in  early  production  as  they  did  on 
development  contracts. 
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Correlates  by  Commodity  Type.  Table  2-12  summarizes  the  significant  variables  that  contribute 
to  total  cost  growth  on  early  production  contracts  for  five  different  system  commodity  types: 
aircraft,  missiles,  munitions,  space  systems,  and  ships.  Again,  it  is  important  to  note  that  other 
variables  listed  in  Table  2-7  (contract  type,  definitization,  and  Military  Departments)  were 
examined  and  found  to  have  no  significant  contribution  statistically  to  total  cost  growth. 


Table  2-12.  Contributors  to  Total  Contract  Cost  Growth  on  Early  Production  Contracts  by 
Commodity  Type  (1970-2011). 


Significant  variables 

Total  contract  cost  growth 
corresponding  effect 

Aircraft 

(n=73) 

Work  content:  each  +1  percentage-point 

+1.08  percentage-points 

Missiles 

(n=136) 

Work  content:  each  +1  percentage-point 

+1.12  percentage-points 

Army  contracts 

+10  percentage-points 

Munitions 

[insufficient  data] 

Space  systems 

(n=36) 

Work  content:  each  +1  percentage-point 

+0.92  percentage-points 

Ships 

(n=86) 

Work  content:  each  +1  percentage-point 

+1.04  percentage-points 

Schedule  growth:  each  +1  percentage-point 

+0.61  percentage-points 

NOTE:  The  variable  in  these  four  regressions  explain  97  percent,  91  percent,  90  percent,  and  96  percent  of  the 
observed  variation  in  total  contract  cost  growth,  respectively.  Other  variables  listed  in  Table  2-7  (schedule,  size, 
definitization,  contract  type,  other  system  commodity  types,  and  other  Military  Departments)  were  examined  but 
found  to  have  no  significant  contribution  statistically  to  total  cost  growth.  Source  cost  data  were  reported  in  "then- 
year"  dollars  (unadjusted  for  inflation). 


As  expected,  the  effect  of  work  content  growth  is  statistically  significant  on  all  regressions,  but 
some  had  a  stronger  relationship  than  others.  We  also  found  other  variables  for  some 
commodity  types  that  also  exhibited  significantly  higher  total  cost  growth  historically. 
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In  missile  production,  Army  early  production  contracts  generally  were  10  percentage-points 
higher  on  total  cost  growth  (all  other  things  equal). 

There  were  insufficient  data  to  test  for  correlates  of  total  cost  growth  on  early  production 
munitions  contracts. 

For  ship  production,  schedule  growth  is  now  a  large  contributor  to  total  cost  growth.  Each 
percentage-point  increase  in  schedule  generally  correlated  with  a  higher  total  cost  by  about 
0.61  percentage  points.  This  correlation  only  showed  for  ship  production,  and  recall  that  the 
only  other  commodity  for  which  schedule  growth  was  a  factor  was  for  missile  production 
contracts  (discussed  earlier). 

Interestingly,  there  was  no  UCA  correlation  with  total  cost  growth  for  ship  production  (recall 
there  was  a  large  UCA  effect  on  development  contracts).  This  result  is  in  line  with  the  other 
UCA  findings  in  the  report:  We  found  no  instances  where  UCAs  (as  used  by  the  DoD  historically) 
correlated  statistically  with  cost  growth  on  these  MDAP  contracts. 

Temporal  Patterns  of  Cost  Growth:  Early  Production 

As  with  development  contracts,  we  also  examined  temporal  cost  data  on  119  large  early 
production  MDAP  contracts  programs  from  1993-2011.  We  found  that  the  same  seven  cost 
growth  profiles  from  development  contracts  covered  the  general  patterns  for  early  production 
contracts.  Examples  from  each  pattern  are  shown  in  the  following  figures. 
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Figure  2-36.  Example  Early  Production  Contract  Cost-Growth  Profiles:  "Well  Managed." 
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Figure  2-37.  Example  Development  Contract  Cost-Growth  Profiles:  "Premature  Start." 
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Figure  2-38.  Example  Development  Contract  Cost-Growth  Profiles:  "Premature  Start  +  Poor 
Estimate." 
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Figure  2-39.  Example  Development  Contract  Cost-Growth  Profiles:  "Work  Added  Later." 


Work  added  later  (a  DDG-51  contract) 


Growth  from 
initial  contract 
budget  base 


Work  content  ,  Total  cost 
growth  growth 


66 


Performance  of  the  Defense  Acquisition  System ,  2013 


Figure  2-40.  Example  Development  Contract  Cost-Growth  Profiles:  "Work  Added  Later  +  Poor 
Estimate." 
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Figure  2-41.  Example  Development  Contract  Cost-Growth  Profiles:  "Poor  Estimate." 
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Figure  2-42.  Example  Development  Contract  Cost-Growth  Profiles:  "Apparent  Poor 
Management." 


Figure  2-43.  Summary  of  Early  Production  Contract  Cost-Growth  Profiles. 
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NOTE:  n=119  total  early  production  contracts. 
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In  production,  half  of  the  major  contracts  (61  out  of  119,  or  51  percent)  were  well  managed 
with  minimal  cost  growth  of  any  kind  (the  ratio  was  lower  at  28  percent  for  development 
contracts).  The  remaining  contracts  had  some  type  of  large  cost  growth  in  one  of  the  six 
categories. 

As  in  development  contracts,  cost-over-target  (interpreted  as  poor  cost  estimates)  was  a 
dominant  pattern  (20  out  of  119  contracts,  or  35  percent),  but  now  large  work  content  added 
early  in  the  contract  (interpreted  as  a  premature  start)  also  was  dominant  at  the  same  level  (20 
contracts;  35  percent). 

Interestingly,  work  content  more  often  was  added  earlier  ("premature  start,"  with  or  without 
cost-over  target  growth)  for  production  contracts,  whereas  in  development  work  content  was 
more  often  added  later. 

Moreover,  only  2  of  119  contracts  (less  than  2  percent)  exhibited  cost  growth  patterns  that  may 
indicate  poor  contract  management  (this  is  a  decrease  from  the  8  percent  for  development 
contract  patterns). 

Early  Contract  Work  Content  Stability:  Production 

As  with  the  development  contract  discussion  above,  we  examined  the  effects  of  early  work 
content  stability  on  long-term  cost  and  schedule  growth.  The  following  are  the  statistical  results 
of  examining  major  MDAP  development  contracts  from  June  1992  through  December  2011. 

•  Early  production_ contracts  with  early  work  content  stability  had  significantly  lower 

■  Total  cost  growth  -85  percentage -points 

■  Work  content  growth  -51  percentage  points 

■  Schedule  growth  -17  percentage -points 

(all  at  the  median  and  all  other  things  equal;  cost  data  unadjusted  for  inflation). 

For  those  patterns  where  work  content  was  stable  in  the  first  year,  we  found  that  cost  and 
schedule  growth  also  were  more  stable.  This  supports  the  view  that  a  well-understood  and 
-defined  contract  at  the  outset  tends  to  perform  better  in  the  long  term.  We  found  no 
statistically  significant  difference  in  costs-over-target  on  early  production  contracts  between 
stable  and  unstable  work  content  in  the  first  year  of  the  contract. 

Effect  of  Contract  Type  Contract  Cost  and  Schedule  Growth:  Early  Production 

As  with  development,  our  statistical  analyses  checked  whether  contract  type  correlated  with 
cost  growth  on  early  production  contracts.  Statistical  analysis  of  the  effect  of  contract  type  on 
early  production  for  440  MDAP  contracts  between  1970  and  2011  is  summarized  in  the 
following  figures  and  tables. 

Unlike  the  results  for  development  contracts,  Aircraft  and  UCA  variables  did  not  statistically 
increase  cost  growth  historically,  and  schedule  growth  now  correlates  with  total  cost  growth  (at 
about  one-tenth  the  rate  that  work-content  cost  growth  does);  if  the  early  production  contract 
was  for  the  Army,  there  is  a  measurable  correlation  of  a  12  percentage-points  increase. 
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Again,  no  statistically  significant  differences  were  found  in  contract  type  in  433  MDAP 
development  contracts  or  440  MDAP  early  production  contracts  in  1970-2011. 


Table  2-13.  Fixed-Price  and  Cost-Type  Contract  Effects  on  Early  Production  Total  Contract  Cost 
Growth  (1970-2011). 


Corresponding  effect  on 

Significant  variables_ Total  contract  cost  growth 

Work  content:  each  +1  percentage-point  +1.07  percentage-points 

Schedule  growth:  each  +1  percentage-point  +0.095  percentage-points 


If  is  an  Army  contract  +12  percentage-points 

Insignificant  variables  (no  statistical 
significant  effects  on  total  contract  cost 
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•  Space 

System 
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Ground  tactical  vehicles 


•  Weapons 

_ •  Other 

•  Army 
Military  •  Navy 
Department  •  Air  Force 

•  DoD  (joint) 


NOTE:  The  three  statistically  significant  variables  in  this  regression  explain  92  percent  of  the  observed  variation  in 
total  contract  cost  growth  for  early  production.  Level  of  significance  is  5  percent  (i.e.,  there  is  only  a  5  percent 
chance  we  cannot  reject  the  value  is  actually  zero).  Source  cost  data  were  reported  in  "then-year"  dollars 
(unadjusted  for  inflation). 
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Table  2-13  shows  the  three  variables  that  are  statistically  correlated  with  total  contract  cost 
growth  along  with  the  other  variables  we  tested.  Note  that  contract  type  did  not  correlate  with 
total  contract  cost  growth. 


Figure  2-44.  Contract-Type  Effects  on  Early  Production  Contract  Cost  Growth  (1970-2011). 
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NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


The  statistical  analysis  verifies  that  there  is  no  statistically  significant  difference  in  Figure  2-44 
between  cost-reimbursable  and  fixed-price  contracts  with  respect  to  cost  growth  on  early 
production  contracts. 
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Figure  2-45.  Contract-type  effects  on  early  production  contract  schedule  growth  (1970-2011). 
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The  statistical  analysis  verifies  that  there  is  no  statistically  significant  difference  in  Figure  2-45 
between  cost-reimbursable  and  fixed-price  contracts  with  respect  to  schedule  growth  on  early 
production  contracts. 

UCA  Effects  on  Cost  Growth:  Early  Production 

In  all  our  analyses  of  contracts  in  1970-2011  (in  total  and  the  more  recent  subset  since  June 
1992).  UCAs  did  not  correlate  with  total  cost  grown  on  early  production.  Recall  that  there  were 
some  correlates  on  development  contracts. 
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Institutional  Analyses:  Military  Departments 

This  section  summarizes  various  acquisition  performance  metrics  analyzed  by  Component, 
predominantly  as  measured  by  contract  cost  and  schedule  growth. 

Not  all  of  the  many  defense  acquisition  functions  performed  by  the  Components  show 
statistically  significant  effects  on  cost  and  schedule.  However,  measuring  them  at  the  macro 
level  and  comparing  their  performance  may  provide  useful  indicators  of  performance  and  their 
possible  causes. 

Nunn-McCurdy  Cost  Breaches 

Again,  we  can  get  a  proxy  measure  of  acquisition  performance  by  looking  at  Nunn-McCurdy 
breach  rates  by  Component.  Figure  2-46  shows  Nunn-McCurdy  breach  counts  by  year  from 
1997  to  2010.  Table  2-14  provides  a  different  analysis  counting  the  number  of  programs  that 
never  breach  against  those  that  have  at  least  one  breach. 


Figure  2-46.  Annual  SAR  Nunn-McCurdy  Breaches  (1997-2010). 
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NOTE:  The  criteria  for  breaches  were  changed  in  NDAA  2006,  affecting  counts  starting  with  2005.  These  plots 
include  multiple  breaches  on  a  single  program  if  cost  changed  from  prior  year.  "Joint"  programs  in  this  plot  are 
those  identified  as  "DoD"  in  AT&L's  database.  Breaches  are  determined  using  "base-year"  dollars  (adjusted  for 
inflation). 


The  graphs  in  Figure  2-46  reflect  the  data  printed  in  the  SAR  and  may  differ  slightly  from 
breaches  reported  individually  to  Congress.  For  example,  in  December  2010,  the  Global  Hawk 
program  only  experienced  a  significant  Nunn-McCurdy  breach  of  22.8  percent,  which  is 
reflected  in  graph,  but  Congress  was  notified  of  a  critical  breach  since  the  draft  Program  Office 
Estimate  was  signaling  increased  cost  above  the  25  percent  threshold.  Similarly,  the  December 
1998  SAR  for  the  SADARM  program  indicated  a  significant  breach,  which  is  included  in  the 
graph,  but  the  program  did  not  report  a  breach  since  it  attributed  the  breach  to  rounding  errors 
in  the  unit  cost  calculation. 
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Table  2-14.  Nunn-McCurdy  MDAP  Breach  Rates  by  Component  (1997-2011). 


Total  #  of 
Programs 

#  of 

Programs 

that 

Breached 

Breach 

Rate 

#  of  Programs 
with  at  Most 
a  Significant 
Breach 

Significant 
Breach  Rate 

#  of 

Programs 
with  a 
Critical 

Breach 

Critical 

Breach 

Rate 

Air  Force 

51 

15 

29% 

2 

4% 

13 

25% 

Army 

45 

18 

40% 

8 

18% 

10 

22% 

Joint 

14 

7 

50% 

1 

7% 

6 

43% 

Navy 

60 

14 

23% 

5 

8% 

9 

15% 

Total 

170 

54 

32% 

16 

9% 

38 

22% 

NOTE:  Analyzed  data  on  active  programs  from  the  1997  SAR  to  the  September  2012  SAR.  Some  programs  breach 
because  of  cancellation  (e.g.,  Land  Warrior)  and  some  programs  canceled  before  Nunn-McCurdy  (e.g.,  VH-71).  If 
program  had  both  a  significant  and  critical  breaches,  it  was  only  included  in  the  "programs  with  critical  breach" 
column.  Not  adjusted  for  quantity  or  other  variances.  "Joint"  programs  in  this  table  are  those  identified  as  "DoD" 
in  AT&L's  database.  Breaches  are  determined  using  "base-year"  dollars  (adjusted  for  inflation). 


Historically,  most  Air  Force  and  joint  MDAPs  that  breach  go  critical  (i.e.,  very  few  remain  at  the 
significant  level).14  Only  half  of  Army  breaches  are  critical,  but  Army  MDAPs  breach  at  a  higher 
rate.  Overall,  Navy  MDAPs  have  the  lowest  breach  rate  and  about  two-fifths  of  those  remain 
below  the  critical  level. 

No  attempt  was  made  to  identify  or  remove  program  breaches  based  on  cause.  Some  programs 
may  breach  because  of  cancellation  (e.g.,  Land  Warrior)  and  some  programs  may  have  been 
canceled  before  their  breach  (e.g.,  VH-71).  If  program  had  both  a  significant  and  critical 
breaches,  it  was  only  included  in  the  "programs  with  critical  breach"  column. 

Mission  Effectiveness  and  Suitability  of  Acquired  Systems:  Military  Departments 

Figure  2-47  plots  the  operational  effectiveness  and  suitability  of  the  systems  acquired  by  the 
Military  Departments  as  assessed  by  the  DoD  Director  of  Operational  Test  and  Evaluation 
(DOT&E)  in  3-year  increments. 


14  Note  that  there  is  a  small  sample  set  of  only  13  Joint  programs  in  this  dataset. 
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Figure  2-47.  Percent  of  MDAPs  by  Military  Department  Rated  as  Effective  and  Suitable  for 
Operation  (1984-2011). 
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Contract  Cost  Growth  and  Service  Acquisition  Executives:  Development 

As  with  the  earlier  plots  for  the  USD(AT&L),  we  conducted  a  number  of  nonstatistical 
examinations  looking  at  various  dimensions  of  cost  and  schedule  growth,  for  both  development 
and  production  contracts  within  each  Military  Department  and  showing  Service  Acquisition 
Executive  (SAE)  tenures.  The  charts  for  development  are  included  below;  early  production 
results  are  included  later  in  this  section. 

Some  isolated  patterns  of  interest  emerged  from  prior  analyses  that  are  apparent  in  the 
following  charts,  but  none  are  statistically  significant  unless  they  are  explicitly  as  such. 


Figure  2-48.  Navy  Development  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992-2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-49.  Air  Force  Development  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992- 

2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-50.  Army  Development  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992-2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 


These  figures  seem  to  imply  visually  some  relationships  between  sitting  SAE  for  major  reviews 
before  these  contracts  and  the  eventual  performance  of  those  contracts.  For  example, 
development  contract  schedule  growth  analysis  shows  that  Navy  growths  appear  to  be  better 
controlled  since  2006  (Etter,  Thackrah,  and  Stackley  eras).  However,  such  visual  observations 
are  subjective  at  this  point  rather  than  tested  statistically,  especially  because  these  plots  do  not 
control  for  other  variables  that  may  be  the  dominate  factors  that  affect  cost  growth  on 
contracts. 

However,  while  many  visual  observations  of  these  figures  are  only  apparent,  some  of  our  other 
statistical  analyses  are  evident  in  these  charts.  For  example,  other  analysis  that  found  total  cost 
growth  for  Air  Force  development  contracts  are  lower  by  a  statistically  significant  amount  since 
2002  can  be  seen  visually  in  these  charts  (i.e.,  since  the  Sambur  era). 
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Cycle  Times:  Development 

In  this  section,  we  report  contract  cycle  times  by  Military  Department.  We  only  found  two 
significant  before/after  period  splits:  Navy  before/after  1980,  and  Air  Force  before/after  1995. 
There  were  no  statistically  significant  before/after  period  splits  in  which  Army  cycle  times  were 
significantly  different. 

Analysis  of  development  cycle  time  across  the  Department  and  by  commodity  types  was 
discussed  earlier  in  this  report. 
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Figure  2-51.  Navy  Development  Contract  Cycle  Time  Trends  (1970-2011). 


Navydevelopment  contract  cycle  times  increased  significantly  after  1980 


Navy  cycle  time  for  Navy  cycle  time  for 
contracts  starting  contracts  starting  after 
before  1980 (n  =  24)  1980 (n  =  123) 


Minimum 

25th  percentile 

2.0 

4.1 

1.5 

4.1 

Median  < 

^  5.3 

6.6 

75th  percentile 

b.3 

- - 

Maximum 

13.6 

15.1 

Upper  bounds  of  1-year  bins 

■  Frequency  distribution  for  Navy  cycle  times  for  contracts  starting  before 
1980  (n  =  24) 

■  Frequency  distribution  for  Navy  cycle  times  for  contracts  starting  1980 
or  later  (n  =  123) 


to 

i_ 

ro 

0> 

> 

c 

< V 


E 


u 


> 

U 


Cycle  time  for  Navy  contracts  Cycle  time  for  Navy  contracts 

starting  before  1980  (n  =  24)  starting  after  1980  (n  =  123) 


NOTE:  Navy  contracts  after  1980  included  DDG  51,  Sea  Wolf,  A-12,  V- 22,  LPD  17,  EFV,  MUOS,  and  DDG  1000.  "Box- 
and-whisker"  charts  (lower-right)  show  minimum,  median,  maximum,  and  quartiles. 
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Figure  2-52.  Air  Force  Development  Contract  Cycle  Time  Trends  (1970-2011). 

Air  Forcedevelopment  contract  cycle  times  increased  significantly  after  1995 
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NOTE:  Air  Force  contracts  after  1995  included  Global  Hawk,  GPS  IIF,  SBIRS  High,  NPOESS,  C17  mods,  C-130  AMP, 
FAB-T;  they  also  executed  under  acquisition  concepts  such  as  Total  System  Performance  Responsibility  (TSPR). 
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Cost  Growth:  Development 

This  section  summarizes  contract-level  data  associated  with  Military  Department  development 
cost  growth.  All  indicated  distinctions  are  statistically  significant. 

Historical  Contract  Results  (1970-2011).  Table  2-15  compares  the  total  cost  growth  outcomes 
on  429  MDAP  development  contracts  from  1970  through  2011. 


Table  2-15.  Military  Department  Performance  on  Development  MDAP  Contracts  (1970- 
2011). 


Variables  (medians): 

Army 
(n  =  97) 

Navy 
(n  =  146) 

Air  Force 
(n  =  179) 

DoD 
(n  =  7) 

Total  cost  growth 

*  44% 

30% 

31% 

*  86% 

Work  content  growth 

*  26% 

9% 

*  19% 

*  27% 

Costs-over-target 

*  9% 

*  8% 

2% 

*  54% 

Schedule  growth 

*  23% 

9% 

12% 

*  23% 

UCA  rate 

41% 

39% 

39% 

43% 

Duration  (years) 

5.7 

6.2 

6.2 

10.0 

*Statisticolly  higher  than  unmarked  values  for  this  variable. 


Trends 

Army 
(n  =  97) 

Navy 
(n  =  146) 

Air  Force 
(n  =  179) 

Total  cost  growth 

Upward 

Work  content  growth 

Upward 

Costs-over-target 

Downward 

Schedule  growth 

Downward 

Duration  (years) 

Upward 

Blank  cells  had  no  measurable  trends. 


NOTE:  All  these  contracts  had  completed  at  least  about  30  percent  of  original  contract  dollar  amount  (55  percent 
of  original  schedule).  Identified  trends  are  those  that  are  statistically  measurable  (with  at  least  a  10  percent  level 
of  significance).  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 
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Historically,  the  Navy  and  Air  Force  had  significant  lower  total  cost  and  schedule  growths 
statistically  than  the  Army  and  Joint  programs  (see  Table  2-15).  The  Navy  controlled  work 
content  the  most,  while  the  Air  Force  had  lower  costs-over-target.  Trend  analysis  showed  the 
Air  Force  was  increasing  total  cost  growth  and  work  content  growth,  the  Navy  was  decreasing 
schedule  growth,  and  the  Army  was  decreasing  costs-over-target. 

Recent  Contract  Results.  Table  2-16  compares  the  total  cost  growth  outcomes  on  170  MDAP 
development  contracts  from  June  1992  to  December  2011. 


Table  2-16.  Contract  Total  Cost  Growth  by  Military  Department  in  Development  (FY1993- 
2011). 


Total  Cost  Growth  (Development  Contracts) 


Army 

Navy 

Air  Force 

(n  =  42) 

(n  =  64) 

(n  =  64) 

Median 

28% 

28% 

34% 

Mean 

92% 

54% 

91% 

NOTE:  Differences  were  tested  and  are  not  statistically  significant.  Medians  are  the  better  measure  of 
central  tendency  for  this  skewed  distribution.  Source  cost  data  were  reported  in  "then-year"  dollars 
(unadjusted  for  inflation). 


The  three  Military  Departments  had  statistically  similar  total  cost  growths  since  June  1992. 

The  distributions  are  skewed  to  the  high  side,  so  medians  provide  a  better  measure  of  central 
tendency  of  the  population  while  the  means  give  a  sense  of  the  effect  of  outliers.  The  apparent 
differences  from  examining  the  arithmetic  means  are  due  to  effects  of  higher  outliers  for  the 
Army  and  Navy.  For  example,  the  highest  growths  for  the  Army  and  Air  Force  were  1,221 
percent  and  862  percent,  compared  to  708  percent  for  the  Navy.  Interestingly,  the  lowest  cost 
growths  for  the  Army  and  Air  Force  were  both  negative  at  -47  percent  and  -36  percent, 
respectively,  compared  to  0  percent  for  Navy.  These  calculations  were  not  weighted  by  contract 
size. 

Recent  Contract  Trends.  In  a  different  regression  analysis,  we  found  a  small  but  measurable 
trend  of  reducing  total  cost  growth  for  the  Army  and  Air  Force.  All  other  things  equal  (i.e., 
controlling  for  other  variables),  Army  and  Air  Force  contracts  from  June  1992  through 
December  2011  were  dropping  (-4  percentage-points  and  -3  percentage  points  every  10  years, 
respectively);  however,  these  trends  only  explained  0.1  percent  each  of  the  variation  in  the 
data  when  controlling  for  the  other  variables  in  Table  2-7,  including  schedule  growth,  size, 
definitization,  contract  type,  commodity  type,  and  the  remaining  Military  Departments  (the 
Army).  The  dominant  statistical  correlate  of  total  cost  growth  was  work  content  growth  (as 
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reflected  in  a  higher  contract  target  cost),  which  explained  96  percent  of  the  variation  in  the 
data.  The  remaining  3.4  percent  of  the  variation  is  not  explained  by  the  variables  examined  and 
is  due  to  other  variables  outside  our  dataset  or  random  noise. 

In  another  (less  sophisticated)  analysis  comparing  the  decades  before  and  since  2002,  the  Army 
and  Air  Force  total  cost  growths  were  lower  by  24  and  21  percentage  points,  respectively,  and 
exhibited  decreasing  trends  compared  to  the  prior  decade.  Army  reduced  costs-over-target  and 
has  a  decreasing  trend;  Air  Force  lowered  schedule  growth  and  has  a  decreasing  trend.  Navy 
showed  no  significant  changes  between  the  two  decades.  Air  Force  aircraft  development  total 
cost  growth  on  contracts  since  2002  were  significantly  higher  by  103  percentage-points  than 
those  in  the  prior  decade.  Note  that  these  values  indicate  the  expected  change  in  percentage- 
points  in  total  cost  with  all  other  things  being  equal  (i.e.,  holding  fixed  other  variables  in  our 
data). 

Schedule  Growth 

Table  2-17  compares  schedule  growth  outcomes  on  170  MDAP  development  contracts  from 
June  1992  to  December  2011. 


Table  2-17.  Military  Department  Schedule  Growth  on  MDAP  Development  Contracts 
(FY1993-2011). 


Schedule  Growth  (Development  Contracts) 


Army 

Navy 

Air  Force 

(n  =  42) 

(n  =  64) 

(n  =  64) 

Median 

*  20% 

6% 

3% 

Mean 

87% 

35% 

77% 

*  Statistically  higher  than  unmarked  values  for  this  variable. 

NOTE:  Medians  are  the  better  measure  of  central  tendency  for  this  skewed  distribution. 

The  Air  Force  and  Navy  each  had  significantly  lower  schedule  growth  compared  to  the  Army 
when  tested  statistically  across  the  population  of  contracts. 

In  a  different  regression  analysis,  we  found  that  30  percent  of  the  variation  in  schedule  growth 
is  due  to  work  content  growth,  where  schedule  growth  rose  by  2.2  percentage-points  for  every 
10  percentage-point  increase  in  work  content  cost  (all  other  things  being  equal).  A  very  small 
but  measurable  2  percent  of  the  variation  in  schedule  growth  is  explained  by  Army  contracts 
being  26  percentage  points  higher  (again,  all  other  things  being  equal).  Still,  68  percent  of  the 
variation  in  schedule  growth  is  unexplained  by  examining  the  variables  in  Table  2-7,  including 
size,  definitization,  contract  type,  commodity  type,  and  the  other  two  Military  Departments 
(Navy  and  Air  Force). 
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As  with  cost  growth  generally,  schedule  growth  distributions  were  skewed  to  the  high  side,  so 
medians  provide  a  better  measure  of  central  tendency  while  the  means  give  a  sense  of  the 
effect  of  outliers.  The  apparent  differences  from  examining  the  arithmetic  means  are  due  to 
effects  of  higher  outliers  for  the  Army  and  Navy.  For  example,  the  highest  growths  for  the  Army 
and  Air  Force  were  582  percent  and  459  percent,  compared  to  238  percent  for  the  Navy.  All 
three  departments  had  at  least  one  contract  with  negative  schedule  growth,  although  their  first 
quartile  was  at  0  percent.  These  calculations  were  not  weighted  by  contract  size. 


Program  Schedules.  Finally,  Figure  2-53  shows  preliminary  analysis  of  program  delays  of  their 
Milestone  C  (MS  C),  comparing  actual  dates  to  the  estimate  made  at  the  original  Milestone  B 
development  baseline.  Milestone  C  is  the  review  at  which  approval  is  made  for  a  program  to 
enter  the  Production  and  Deployment  phase.  Note  that  the  data  were  incomplete:  Not  all 
MDAPs  reported  forecasts  of  their  MS  C  dates  at  MS  B,  and  many  MDAPs  have  yet  to  have  their 
MS  C.  Further  analysis  is  needed. 

Figure  2-53.  Milestone  C  Delays  from  Development  Baseline  (1997-2011). 


#  past  MS  C 
Total  Reporting 

Programs  Schedule  Data 


Air  Force 

48 

13 

27% 

Army 

45 

15 

33% 

Joint 

13 

0 

0% 

Navy 

58 

21 

36% 

Total 

164 

49 

30% 

12 

10 

8 

6 

4 

2 

0 


Median  Delays: 

Air  Force  ~  669  days 
Army  ~  365  days 
Navy  ~  181  days 
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Contract  Cost  Growth  and  Service  Acquisition  Executives:  Early  Production 

Below  are  the  early  production  charts  looking  at  various  dimensions  of  cost  and  schedule 
growth,  for  both  development  and  production  contracts  within  each  Military  Department  and 
Service  Acquisition  Executive  (SAE).  Some  isolated  patterns  of  interest  emerged  from  prior 
analyses  that  are  apparent  in  the  following  charts.  Unless  shown,  there  was  no  noteworthy 
pattern  observed. 


Figure  2-54.  Navy  Early  Production  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992-2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-55.  Air  Force  Early  Production  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992- 

2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 
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Figure  2-56.  Army  Early  Production  Contract  Total  Cost  Growth  and  SAE  Tenures  (1992-2011). 
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NOTE:  Normally,  contract  start  dates  should  be  relatively  close  to  prior  major  reviews  (usually  Milestone  B 
decisions)  by  the  SAE  and  MDA  to  approve  contract  award.  Diamonds  on  the  charts  indicate  growths  mostly 
attributable  to  the  start  of  the  contract;  circles  represent  significant  work  added  later.  Any  white  bars  between  SAE 
shaded  regions  represent  periods  where  there  was  no  confirmed  executive.  Source  cost  data  were  reported  in 
"then-year"  dollars  (unadjusted  for  inflation). 


These  figures  seem  to  imply  visually  some  relationships  between  SAE  in  place  for  major  reviews 
before  these  contracts  and  the  eventual  performance  of  those  contracts.  For  example,  since 
2008  (Thackrah  and  Stackley  eras),  Navy  production  total  cost  growth  appears  better  controlled 
than  earlier  Navy  contracts.  However,  all  such  visual  observations  are  subjective  at  this  point 
rather  than  tested  statistically,  especially  because  these  plots  do  not  control  for  other  variables 
that  may  be  the  dominate  factors  that  affect  cost  growth  on  contracts. 

Cost  Growth:  Early  Production 

This  section  summarizes  contract-level  data  associated  with  Military  Department  early 
production  cost  growth.  All  cited  distinctions  are  statistically  significant. 

Historical  Contract  Results  (1970-2011).  Table  2-18  compares  the  total  cost  growth  outcomes 
on  429  MDAP  early  production  contracts  from  1970  through  2011. 
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Table  2-18.  Military  Department  Performance  on  Early  Production  MDAP  Contracts  (1970- 
2011). 


Variables  (medians): 

Army 
(n  =  71) 

Navy 
(n  =  199) 

Air  Force 
(n  =  168) 

Total  cost  growth 

*  14% 

13% 

9% 

Work  content 
growth 

5% 

4% 

4% 

Costs-over-target 

*  2% 

*  4% 

0% 

Schedule  growth 

*  20% 

8% 

*  20% 

UCA  rate 

7% 

38% 

22% 

Duration  (years) 

3.8 

*  5.3 

*4.5 

*Statistically  higher  than  unmarked  values  for  this  variable. 


Trends 

Total  cost  growth 

Work  content 
growth 

Costs-over-target 
Schedule  growth 
Duration  (years) 

NOTE:  Regression  explains  92  percent  of  the  observed  variation  in  total  cost  growth.  Source  cost  data  were 
reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Army  Navy  Air  Force 

(n  =  71)  (n  =  199)  (n  =  168) 


Upward 

Blank  cells  had  no  measurable  trends. 


From  1970-2011,  the  Navy  and  Air  Force  had  significantly  lower  total  cost  growth  than  the 
Army  programs  (see  Table  2-18).  The  Air  Force  had  lower  costs-over-target,  and  there  was  no 
difference  in  controlling  work  content.  The  Navy  controlled  schedule  growth  the  most.  No 
statistically  significant  historical  trends  were  observed  in  cost  and  schedule  growth. 
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Recent  Contract  Results  and  Trends.  In  a  different  regression  analysis,  we  found  that  with  all 
other  things  equal  (i.e.,  controlling  for  other  variables),  Army  contracts  from  June  1992  through 
December  2011  were  53  percentage-points  higher;  however,  this  Army  effect  only  explains  1 
percent  of  the  variation  in  the  data.  In  other  words,  the  effect  is  large  and  measurable,  but 
nearly  all  the  differences  between  cost  growth  on  different  contracts  in  the  data  are  due  to 
variables  other  than  whether  this  was  an  Army  contract.  This  comparison  is  important  because 
we  are  trying  to  look  for  performance  differences  between  institutions  and  processes  as  we 
search  for  lessons  learned.  The  statistics  tell  us  that  even  if  we  were  able  to  eliminate  this  Army 
contract  effect,  the  other  dominant  factors  that  cause  variation  in  cost  growth  still  would  be 
present  and  lead  to  increases  from  the  central  tendency  unless  they  also  were  eliminated. 

Also,  we  found  an  even  smaller  but  measurable  trend  of  increasing  total  cost  growth  for  the 
Navy.  Navy  contract  total  cost  growths  were  increasing  4  percentage-points  every  10  years, 
respectively;  however,  this  trend  only  explained  an  even  smaller  0.1  percent  of  the  variation  in 
the  data.  This  analysis  controlled  for  the  other  variables  in  Table  2-7,  including  schedule  growth, 
size,  definitization,  contract  type,  commodity  type,  and  the  remaining  Military  Departments 
(the  Army).  The  dominant  statistical  correlate  of  total  cost  growth  was  work  content  growth  (as 
reflected  in  a  higher  contract  target  cost),  which  explained  95  percent  of  the  variation  in  the 
data.  The  remaining  3  percent  of  the  variation  is  not  explained  by  the  variables  examined  and  is 
due  to  other  variables  outside  our  dataset  or  random  noise. 

Institutional  Analyses:  Acquisition  Commands  by  Commodity  Type 

We  also  conducted  analyses  of  the  outcomes  on  contracts  by  the  Military  Departments' 
acquisition  commands.  Our  data  do  not  identify  acquisition  commands  directly,  so  we  used  a 
combination  of  commodity  type  and  Military  Department  identifiers  to  infer  acquisition 
commands.  Thus,  in  some  cases  we  had  insufficient  sample  size  and  therefore  are  not 
discussed. 

Aircraft:  Development 

Table  2-19  compares  the  relative  outcomes  of  contracts  nominally  by  the  U.S.  Naval  Air  Systems 
Command  (NAVAIR)  and  the  Air  Force's  Aeronautical  Systems  Center  (ASC)  on  41  MDAP 
development  contracts  from  1970  to  2011. 
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Table  2-19.  Relative  Performance  of  NAVAIR  and  ASC  on  Aircraft  Development  MDAP 
Contracts  (1970-2011). 


Variables  (medians): 

NAVAIR 
(n  =  21) 

ASC 
(n  =  20) 

Total  cost  growth 

40% 

*  67% 

Work  content 
growth 

9% 

*  37% 

Costs-over-target 

16% 

19% 

Schedule  growth 

5% 

*  29% 

UCA  rate 

29% 

35% 

Duration  (years) 

6.7 

7.7 

*Statistically  higher  than  unmarked  values  for  this  variable. 


Trends 

Total  cost 

growth 

Work  content 

growth 

Costs-over- 

target 

Schedule 

growth 

Duration  (years) 


NAVAIR  ASC 

(n  =  21)  (n  =  20) 


Downward 

Upward 

Blank  cells  had  no  measurable  trends. 

NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


NAVAIR  controlled  total  cost  growth  better  than  the  ASC  by  controlling  work  content  better, 
though  costs-over-target  were  similar.  NAVAIR  controlled  schedule  growth  better,  and  ASC  has 
a  long-term  trend  of  decreasing  schedule  growth  on  aircraft  contracts. 


This  highlights  a  difference  at  the  acquisition  command  level  that  was  not  apparent  in  the 
Military  Department  level  data.  NAVAIR  shows  more  favorable  results  because  it  controls  work 
content  better  than  ASC.  This  is  one  instance  where  we  are  beginning  to  use  data  analysis  to 
explain  why  some  institutions  perform  better  than  others.  We  still  do  not  know  how  NAVAIR 
controlled  work  content  more,  but  that  control  is  why  NAVAIR  performs  better  on  aircraft 
developments. 
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Missiles:  Development 

The  Army  Aviation  and  Missile  Life  Cycle  Management  Command  (AMCOM),  NAVAIR,  and  the 
Air  Force's  Air  Armament  Center  (AAC)  were  analyzed  next. 

Table  2-20.  Relative  Performance  of  AMCOM,  NAVAIR,  and  AAC  on  Missile  Development 
MDAP  Contracts  (1970-2011). 


Variables  (medians): 

AMCOM 
(n  =  32) 

NAVAIR 
(n  =  51) 

AAC 
(n  =  74) 

Total  cost  growth 

**  86% 

*  26% 

21% 

Work  content  growth 

**  25% 

9% 

15% 

Costs-over-target 

**  21% 

*  4% 

0% 

Schedule  growth 

**  34% 

4% 

*  11% 

UCA  rate 

47% 

31% 

22% 

Duration  (years) 

6.7 

5.0 

5.4 

**Statistically  higher  than  all  other  values  for  this  variable. 

*  Statistically  higher  than  unmarked  values  for  this  variable. 


Trends 

Total  cost 

growth 

Work  content 

growth 

Costs-over- 

target 

Schedule 

growth 

Duration 

(years) 


AMCOM  NAVAIR  AAC 

(n  =  32)  (n  =  51)  (n  =  74) 


Downward 

Downward 

Downward 

Downward 

Blank  cells  had  no  measurable  trends. 

NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 
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AAC  controlled  total  cost  growth  the  most,  followed  by  NAVAIR;  AAC  and  NAVAIR  both 
controlled  work  content  better  than  AMCOM.  NAVAIR  controlled  schedule  growth  the  most, 
followed  by  AAC.  AAC  controlled  costs-over-target  the  most,  followed  by  NAVAIR.  Despite  being 
higher,  AMCOM  showed  improving  trends  in  total  cost  growth,  work  content  growth,  and  cost- 
over-target.  NAVAIR  had  an  improving  trend  in  schedule  growth.  Also,  recall  that  in  the  prior 
analysis  of  missile  development  contracts  (see  Table  2-9,  wherein  Army  (AMCOM)  total  cost 
growth  generally  was  30  points  higher,  all  other  things  equal  [i.e.,  Army's  median  total  cost 
growth  was  higher  still  for  other  reasons]). 

Again,  this  level  of  analysis  highlights  a  difference  at  the  acquisition  command  level  that  was 
not  apparent  in  the  Military  Department  level  data.  AAC  and  (to  a  lesser  extent)  NAVAIR  show 
more  favorable  results  mostly  because  they  control  work  content  (and,  to  a  lesser  extent, 
schedule  growth)  better  than  AMCOM.  This  is  another  instance  where  we  are  beginning  to  use 
data  analysis  to  explain  why  some  institutions  perform  better  than  others.  We  still  do  not  know 
how  AAC  and  NAVAIR  controlled  work  content  more,  but  that  control  is  why  they  perform 
better  on  missile  developments. 

Munitions:  Development 

Munitions  included  the  Army's  Joint  Munitions  and  Lethality  Life  Cycle  Management  Command 
(JM&LLCMC),  NAVAIR,  and  the  Air  Force's  AAC. 
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Table  2-21.  Relative  Performance  of  JM&LLCMC,  NAVSEA,  and  AAC  on  Munitions 
Development  MDAP  Contracts  (1970-2011). 


JM&LLCMC 

NAVSEA 

AAC 

Variables  (medians): 

(n  =  7) 

(n  =  2) 

(n  =  11) 

Total  cost  growth 

33% 

N/A 

31% 

Work  content 
growth 

*  55% 

N/A 

25% 

Costs-over-target 

0% 

N/A 

0% 

Schedule  growth 

23% 

N/A 

38% 

UCA  rate 

43% 

N/A 

64% 

Duration  (years) 

7.4 

N/A 

5.6 

* Statistically  higher  than  AAC's  values  for  this  variable. 


Trends 

JM&LLCMC 

NAVSEA 

AAC 

(n  =  7) 

(n  =  2) 

(n  =  11) 

Total  cost  growth 

N/A 

Work  content 
growth 

N/A 

Costs-over-target 

N/A 

Downward 

Schedule  growth 

N/A 

Duration  (years) 

N/A 

Blank  cells  had  no  measurable  trends. 

NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 

AAC  controlled  work  content  more  than  the  Army's  JM&LLCMC.  Also,  AAC  showed  an 
improving  trend  on  costs-over-target.  NAVSEA  did  not  have  a  large  enough  sample  size  to 
analyze. 

This  highlights  a  further  difference  at  the  acquisition  command  level  that  was  not  apparent  in 
the  Military  Department  level  data.  Here  the  effect  is  somewhat  weaker  in  that  work  content 
only  explained  69  percent  of  the  variation  in  the  cost  growth  data,  but  it  still  indicates  that  AAC 
showed  a  more  favorable  result  mostly  because  it  controls  work  content  better  than 
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JM&LLCMC.  Again,  data  analysis  is  helping  us  to  begin  explaining  why  some  institutions  perform 
better  than  others.  We  still  do  not  know  how  AAC  controlled  work  content  more,  but  that 
control  is  why  they  perform  better  on  munitions  developments. 

Aircraft:  Early  Production 

Table  2-22  compares  the  relative  outcomes  of  contracts  nominally  by  NAVAIR  and  ASC  on  73 
MDAP  early  production  contracts  from  June  1992  to  December  2011. 

Table  2-22.  Relative  Performance  of  NAVAIR  and  ASC  on  Aircraft  Early  Production  MDAP 
Contracts  (1970-2011). 

Aircraft  Early  Production  (n  =  73) 

Variables  (medians): 

Total  cost  growth 

Work  content  growth 

Costs-over-target 

Schedule  growth 

UCA  rate 

Duration  (years) 

*Statistically  higher  than  unmarked  values  for  this  variable. 

Aircraft  Early  Production  (n  =  73) 

Trends 

Total  cost  growth 

Work  content 
growth 

Costs-over-target 
Schedule  growth 
Duration  (years) 

Blank  cells  had  no  measurable  trends. 

NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


NAVAIR  ASC 

(n  =  29)  (n  =  44) 


Upward 

Upward 

NAVAIR  ASC 


CM 

II 

c 

(n  =  44) 

2% 

*  8% 

1% 

*  4% 

*  0% 

0% 

7% 

*  38% 

55% 

27% 

4.3 

4.5 
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NAVAIR  controlled  total  cost  growth  better  than  ASC  by  controlling  work  content  better, 
although  costs-over-target  were  measurably  higher  on  NAVAIR  aircraft  contracts.  As  in 
development,  NAVAIR  controlled  schedule  growth  better.  ASC  has  a  statistically  significant  long¬ 
term  trend  of  larger  costs-over-target  on  aircraft  contracts. 

As  we  found  in  analyzing  development  contracts,  this  highlights  a  difference  at  the  acquisition 
command  level  that  was  not  apparent  in  the  Military  Department  level  data.  NAVAIR  showed  a 
more  favorable  result  because  it  controls  work  content  better  than  ASC. 

Missiles:  Early  Production 

The  Army  Aviation  and  Missile  Life  Cycle  Management  Command  (AMCOM),  NAVAIR,  and  the 
Air  Force's  Air  Armament  Center  (AAC)  were  analyzed  next. 
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Table  2-23.  Relative  Performance  of  JM&LLCMC,  NAVSEA,  and  AAC  on  Missile  Early 
Production  MDAP  Contracts  (1970-2011). 


Missile  Early  Production  (n  =  136) 


Variables  (medians): 

AMCOM 
(n  =  30) 

NAVAIR 
(n  =  42) 

AAC 
(n  =  64) 

Total  cost  growth 

*  10% 

2% 

7% 

Work  content 
growth 

\o 

ON 

00 

* 

1% 

2% 

Costs-over-target 

*  1% 

*  0% 

-2% 

Schedule  growth 

*  20% 

7% 

0% 

UCA  rate 

3% 

24% 

16% 

Duration  (years) 

4.1 

3.8 

4.0 

*Statistically  higher  than  unmarked  values  for  this  variable. 


Missile  Early  Production  (n  =  136) 


Trends 

AMCOM 

NAVAIR 

AAC 

(n  =  30) 

(n  =  42) 

(n  =  64) 

Total  cost 
growth 

Downward 

Work  content 
growth 

Downward 

Costs-over- 

target 

Schedule  growth 

Downward 

Duration  (years) 

Blank  cells  had  no  measurable  trends. 


NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 

NAVAIR  and  AAC  both  were  lower  than  AMCOM  in  total  cost  and  work  content;  AAC  was  lower 
than  both  AMCOM  and  NAVAIR  on  costs-over-target.  NAVAIR  and  AAC  were  both  lower  than 
AMCOM  in  schedule  growth.  NAVAIR  had  an  improving  trend  in  total  cost  and  work  content 
growth  and  AMCOM  showed  an  improving  trend  in  schedule  growth.  Again,  this  level  of 
analysis  indicates  that  NAVAIR  and  AAC  show  more  favorable  results  predominantly  because 
they  control  work  content  better  than  AMCOM. 
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Institutional  Analyses:  Prime  Contractors 

This  section  summarizes  a  comparison  of  cost  and  schedule  growth  among  the  major  prime 
defense  contractors  on  development  and  early  production  MDAP  contracts.  Table  2-24  shows 
the  groupings  employed  to  deal  with  consolidation  of  legacy  companies  into  the  current 
defense  prime  contractors  cited.  Here  we  broke  out  five  major  primes  and  grouped  the  others 
into  a  single  set. 

Table  2-25  and  Table  2-26  show  a  breakout  of  the  number  of  MDAP  development  and  early 
production  contracts,  respectively,  including  the  share  (fraction)  by  number  and  by  dollar  value. 

Table  2-24,  Major  Prime  Contractor  Groupings  after  Consolidation. 


"Lockheed  Martin"  included  contracts  with  the  following  companies: 

Gould 

Marinette  (for  shipbuilding) 

Lockheed 

Martin  Marietta 

Longbow  LLC  (joint  venture  with  Northrop  MEADs  International 

Grumman) 

Sperry 

"Boeing"  included  contracts  with  the  following  companies: 

Hughes  Helicopter 

McDonnell  Douglas 

Rockwell 

"Northrop  Grumman"  included  contracts  with  the  following  companies: 

Avondale  (shipbuilding) 

Litton 

Grumman 

Newport  News 

Huntington  Ingalls  (spin-off  for  ship  construction)  Northrop 

Intermarine  (shipbuilding) 

Tenneco  (shipbuilding) 

"General  Dynamics"  included  contracts  with  the  following  companies: 

Bath  Iron  Works 

GMGDLS 

Electric  Boat 

NASSCO  (shipbuilding) 

"Raytheon"  included  contracts  with  the  following  company: 

Hughes  Aircraft  (missiles) 

"Others"  included  contracts  with  the  following  companies: 

Aerojet 

Garrett 

LTV 

Alliant  (including  Honeywell  propulsion) 

General  Atomics 

MDTT 

AT&T 

General  Electric 

Texas  Instruments 

Austal 

GTE 

Thiokol 

Autonetics 

Honeywell  (engines) 

TRW 

AVCO 

Hercules 

Unisys 

Bell  Textron 

IBM 

United  Technologies  (including 

Eaton 

Interstate  Electronics 

Sikorsky  and  Pratt  &  Whitney) 

Fairchild 

LHTEC 

Vought 

FMC 

Logicon 

Westinghouse 

Ford  Communications  and  Aerospace 

Loral 

Williams  International 
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Table  2-25.  Data  on  Prime  Contractor  Shares  of  Development  Contracts  (1994-2011). 


Development  5  hares  (1994-2011),  n  -  157 


Company 

Number  of  MDAP 

contracts 

S  hare  of  MDAP 
contracts  (by  number) 

5  hare  of  MDAP 
contracts  by  $  value 

Lockheed  Martin 

40 

25% 

42%  t 

Boeing 

28 

18% 

19%  i 

Northrop  G  rumman 

26 

17% 

19%  T 

G  eneral  Dynamics 

14 

9% 

4%  l 

R  aytheon 

21 

13% 

5%  t 

All  others 

28 

18% 

12%  1 

Total 

157 

100% 

100% 

NOTE:  Red  arrows  indicate  direction  of  change  in  share  value  since  1994. 

Table  2-26.  Data  on  Prime  Contractor  Shares  of  Early  Production  Contracts  (1994-2011). 


Early  Production  Shares  (1994-2011),  n  -  100 


Company 

Number  of  MDAP 

contracts 

5  hare  of  MDAP 
contracts  (by  number) 

S  hare  of  MDAP 
contracts  by  $  value 

Lockheed  Martin 

14 

14% 

18%  t 

Boeing 

20 

20% 

22%  1 

Northrop  G  rumman 

31 

31% 

26%  t 

G eneral  Dynamics 

12 

12% 

21% 

R  aytheon 

5 

5% 

1%  1 

All  others 

18 

18% 

12% 

Total 

100 

100% 

100% 

NOTE:  Red  arrows  indicate  direction  of  change  in  share  value  since  1994. 


We  used  FY1994  as  the  start  point  for  this  analysis  because  it  began  a  new  period  of  industry 
consolidation.15  We  analyzed  157  development  contracts  and  100  early  production  contracts 
from  FY1994-2011.  Additional  analysis  was  conducted  to  assess  relative  performance  among 
prime  contractors  for  various  commodity  types. 

The  only  two  breakdowns  that  contained  sufficient  data  for  statistical  analysis  after  partitioning 
by  both  commodity  type  and  prime  contractors  were  aircraft  (development  and  early 
production)  and  ships  (early  production  only.  No  difference  was  found  in  ship  development 
contracts  due  to  the  small  sample  size). 


Deputy  Secretary  of  Defense  William  Perry  told  industry  in  the  fall  of  1993  that  budget  cutbacks  will  require 
major  consolidations.  Thus,  1994  would  be  an  appropriate  starting  point  when  examining  industry  performance. 
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Cost  and  Schedule:  Development  Contracts 

Table  2-27  shows  the  cost  and  schedule  outcomes  by  prime  contractor  for  all  development 
contracts  from  FY1994  to  December  2011. 

Table  2-27.  Cost  and  Schedule  Growth  by  Primes  on  DoD-Wide  Development  Contracts 
(1994-2011). 


Development  Contracts  (medians) 


Lockheed 

Boeing 

Northrop 

General 

Raytheon 

All 

Total 

Variables: 

Martin 

Grumman 

Dynamics 

Others 

sample 

Total  cost 
growth 

**  42% 

**  46% 

**  31% 

**  36% 

*  25% 

16% 

28% 

Schedule 

growth 

1% 

**  17% 

*  6% 

*  6% 

0.0% 

0.0% 

4% 

**Statistically  higher  than  all  other  values  for  this  variable  (not  considering  total  sample  column). 

*  Statistically  higher  than  unmarked  values  for  this  variable  (not  considering  total  sample  column). 
NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Most  of  the  major  primes  except  Raytheon  (Lockheed  Martin,  Boeing,  Northrop  Grumman,  and 
General  Dynamics)  had  statistically  similarly  total  cost  growth  on  development  contracts. 
Raytheon  was  lower,  and  the  remaining  primes  (as  a  group)  were  lower  than  Raytheon.  All 
values  shown  are  medians. 

For  schedule  growth,  Boeing  had  the  highest  statistically,  followed  by  Northrop  Grumman  and 
General  Dynamics.  Lockheed,  Raytheon,  and  all  others  (as  a  group)  were  the  lowest.  All  values 
shown  are  medians. 

Table  2-28  now  examines  just  contracts  for  aircraft  development  by  the  three  prime 
contractors. 


Table  2-28.  Cost  and  Schedule  Growth  by  Primes  on  Aircraft  Developments  (1994-2011). 


Program_ Contractor 


Aircraft  Development  Contracts  (medians) 

F35 

Lockheed  Martin 

C-130J  Block  Upgrade 

Lockheed  Martin 

Lockheed 

Martin 

Boeing 

Northrop 

Grumman 

Total 

sample 

MH-60R 

Lockheed  Martin 

Variables: 

MV-22 

Boeing 

Apache  Block  3 

Boeing 

Total  cost 
growth 

P8A 

Boeing 

56% 

52% 

*  162% 

72% 

Comanche 

Boeing 

UAS  BAMS 

Northrop  Grumman 

Schedule 

growth 

Global  Hawk 

Northrop  Grumman 

-9% 

*  39% 

4% 

7% 

* Statistically  higher  than  all  other  values  for  this  type  of  growth  (not  considering  total  sample  column). 

NOTE:  The  table  on  the  right  lists  the  programs  by  name  and  contractor.  Source  cost  data  were  reported  in  "then- 
year"  dollars  (unadjusted  for  inflation). 
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Analyzing  just  aircraft  development  contracts,  Northrop  Grumman  had  significantly  higher  total 
cost  growth  than  Lockheed  Martin  and  Boeing. 

As  for  schedule  growth,  Boeing  had  the  highest.  Lockheed  Martin's  median  schedule  growth, 
while  negative,  is  not  significantly  different  that  Northrop  Grumman's  value. 

Table  2-29  now  examines  just  contracts  for  ship  development  by  the  two  prime  contractors. 


Table  2-29.  Cost  and  Schedule  Growth  by  Primes  on  Ship  Development  (1994-2011). 


Ship  Development  Contracts  (medians) 


Northrop 

General 

Total 

Variables: 

Grumman 

Dynamics 

sample 

Total  cost 

42% 

45% 

49% 

growth 

Schedule 

22% 

47% 

27% 

growth 

Whether  the  differences  in  these  values  are 
statistically  significance  could  not  be  tested  due  to 
low  sample  size. 


Program 

Contractor 

CVN  78 

Northrop  Grumman 

LHA  6 

Northrop  Grumman 

CVN  21  CP  FY05 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LPD 

Northrop  Grumman 

LCS 

General  Dynamics 

LCS 

General  Dynamics 

DDG  1000 

General  Dynamics 

DD(X) 

General  Dynamics 

NOTE:  The  table  on  the  right  lists  the  programs  by  name  and  contractor.  Source  cost  data  were  reported  in  "then- 
year"  dollars  (unadjusted  for  inflation). 


As  mentioned  above,  for  ship  development,  we  found  no  statistically  significant  differences 
among  the  major  primes  due  to  the  small  sample  size. 

Cost  and  Schedule:  Early  Production  Contracts 

As  compared  to  the  analysis  for  development  contracts,  the  production  data  shows  a  partial 
shift  in  total  cost  growth  to  the  pool  of  non-major  primes  and  shifts  in  abilities  to  meet 
schedules  (see  Table  2-30). 
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Table  2-30.  Cost  and  Schedule  Growth  by  Primes  on  DoD-Wide  Early  Productions  (1994- 
2011). 

Early  Production  Contracts  (medians) 


Lockheed 

Boeing 

Northrop 

General 

Raytheon 

All 

Total 

Variables: 

Martin 

Grumman 

Dynamics 

Others 

sample 

Total  cost 
growth 

*  8% 

0.3% 

2% 

**  15% 

**  16% 

**  37% 

9% 

Schedule 

growth 

3% 

*  15% 

0.0% 

2% 

**  29% 

1% 

0.0% 

**Statistically  higher  than  all  other  values  for  this  variable  (not  considering  total  sample  column). 

*  Statistically  higher  than  unmarked  values  for  this  variable  (not  considering  total  sample  column). 
NOTE:  Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Three  of  the  five  major  primes  had  lower  total  cost  growth  than  the  pool  of  remaining 
contractors  along  with  General  Dynamics  and  Raytheon.  Lockheed  Martin  was  better;  Boeing 
and  Northrop  Grumman  were  better  still. 

Breaking  down  total  cost  growth  into  its  constituents,  the  pool  of  "All  Other"  prime  contractors 
had  the  highest  work  content  growth.  Lockheed  Martin  and  Northrop  Grumman  were  lower 
and  statistically  similar;  Boeing,  General  Dynamics,  and  Raytheon's  levels  were  similar  among 
themselves.  On  cost-over-target.  General  Dynamics  and  the  "All  Other"  group  were  the 
highest,  followed  by  Lockheed  Martin.  Boeing,  Northrop  Grumman,  and  Raytheon  were  lower 
at  statistically  similar  values. 

As  for  schedule  growth,  Raytheon  was  the  highest,  followed  by  Boeing.  Differences  between 
the  lower  Lockheed  Martin,  General  Dynamics,  and  "All  Others"  medians  were  not  significant. 

Table  2-31  now  examines  just  contracts  for  early  production  aircraft  by  the  three  prime 
contractors. 
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Table  2-31.  Cost  and  Schedule  Growth  by  Primes  on  Early  Production  Aircraft  (1994-2011). 


Aircraft  Early  Production  Contracts  (medians) 


Lockheed 

Boeing 

Northrop 

Total 

Variables: 

Martin 

Grumman 

sample 

Total  cost 
growth 

*  9.5% 

0.0% 

2% 

4% 

Schedule 

growth 

0.0% 

17% 

39% 

14% 

*Statistically  higher  than  all  other  values  for  this  variable  (not  considering  total 
sample  column). 

NOTE:  The  table  on  the  right  lists  the  programs  by  name  and  contractor.  Source 
cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for  inflation). 


Program 

Contractor 

F35 

Lockheed  Martin 

F35 

Lockheed  Martin 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

V22 

Boeing 

CH47 

Boeing 

Ch47 

Boeing 

F/A18 

Boeing 

F/A18 

Boeing 

F/A18 

Boeing 

F/A18 

Boeing 

E2D 

Northrop  Grumman 

E2D 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

Global  Hawk 

Northrop  Grumman 

For  aircraft  early  production,  Lockheed  Martin  had  higher  total  cost  growth  than  Boeing  and 
Northrop  Grumman,  driven  by  costs-over-target  on  the  first  two  F-35  low-rate  initial  production 
(LRIP)  contracts.  In  this  case,  further  analysis  revealed  that  cost-over-target  was  the  major 
contributor  to  total  cost  growth  on  LRIP  1  and  2.  Work  content  on  these  two  contracts  was 
stable  (and,  in  fact,  slightly  negative),  as  it  was  on  the  Boeing  and  Northrop  Grumman  aircraft 
early  production  contracts. 

Differences  in  schedule  growth  are  not  statistically  significant  for  aircraft  early  production. 

Table  2-32  now  examines  just  contracts  for  early  production  ships  by  the  two  prime 
contractors. 
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Table  2-32.  Cost  and  Schedule  Growth  by  Primes  on  Early  Production  Ships  (1994-2011). 


Ship  Development  Contracts  (medians) 


Northrop 

General 

Total 

Variables: 

Grumman 

Dynamics 

sample 

Total  cost 

27% 

24% 

27% 

growth 

Schedule 

0.0% 

5% 

0.0% 

growth 

Differences  are  not  statistically  significant. 

NOTE:  The  table  on  the  right  lists  the  programs  by  name  and  contractor. 
Source  cost  data  were  reported  in  "then-year"  dollars  (unadjusted  for 
inflation). 


Program 

Contractor 

Arleigh  Burke 

Northrop  Grumman 

Arleigh  Burke 

Northrop  Grumman 

Arleigh  Burke 

Northrop  Grumman 

CVN  77 

Northrop  Grumman 

LHD  7 

Northrop  Grumman 

Arleigh  Burke 

General  Dynamics 

Arleigh  Burke 

General  Dynamics 

Arleigh  Burke 

General  Dynamics 

T-AKE 

General  Dynamics 

T-AKE 

General  Dynamics 

T-AKE 

General  Dynamics 

SSN  778 

General  Dynamics 

SSN  779 

General  Dynamics 

For  ship  early  production,  the  total  cost  growth  and  schedule  growth  differences  are  not 
statistically  significant. 

This  analysis  will  be  extended  and  expanded  in  subsequent  reports. 
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3.  Needs  for  Additional  Analysis 

Policies  and  Processes  for  Further  Analysis 

This  report  is  just  the  beginning.  We  do  not  have  all  the  answers,  but  it  is  important  to  share 
what  we  do  know  and  start  down  the  road  for  further  progress  toward  understanding 
performance  and  answering  those  questions. 

Many  policies  and  processes  have  changed  in  past  attempts  to  improve  acquisition.  Our  efforts 
to  pursue  data-driven  analysis  are  building  a  capacity  to  more  clearly  understand  the 
effectiveness  of  these  policy  changes  so  we  can  keep  those  that  work  and  replace  those  that  do 
not.  We  recognize  the  need  to  focus  further  efforts  on  candidates  that  we  hypothesize  may  be 
major  contributors  for  change.  The  following  list  illustrates  some  candidates  for  which  the 
Department  is  pursuing  data  and  analytic  approaches: 

•  Cost  growth  on  MDAP  sustainment  costs 

•  Cycle  time  trends  and  effects  on  decision  quality  and  meeting  mission  needs 

•  Meeting  key  system  performance  requirements 

•  Setting  Preliminary  Design  Review  (PDR)  before  Milestone  B 

•  Competitive  Prototyping  for  Risk  Reduction,  not  Proof  of  Concept 

•  Reducing  technical  risks  during  Technology  Development  phase 

•  Cost  and  schedule  growth  relationships  between  contract-level  and  program-level 

•  Better  Buying  Power  (e.g.,  should-cost;  see  below) 

Better  Buying  Power 

AT&L  is  instituting  a  series  of  continuous  improvement  initiatives  under  the  Better  Buying 
Power  (BBP)  effort16  to  improve  the  performance  of  the  defense  acquisition  system  to  improve 
efficiency  and  cost-saving  processes.  Implementation  of  these  efforts  continues,  and  we  are 
pursuing  data  collection  activities  to  enable  analysis  of  whether  and  to  what  degree  these 
initiatives  improve  acquisition  performance.  Version  2.0  of  Better  Buying  Power  was 
implemented  in  April  2013  and  continues  and  expands  upon  initiatives  begun  in  2010.  Version 
2.0  has  seven  focus  areas,  including  a  new  one  on  the  importance  of  our  acquisition  workforce. 


16  See  http://bbp.dau.mil/. 


105 


Performance  of  the  Defense  Acquisition  System ,  2013 


Further,  Better  Buying  Power  includes  a  strong  emphasis  on  data  collection  and  analysis  to 
understand  the  effectiveness  of  the  policies  and  processes  addressed  both  within  the  BBP 
initiatives,  and  more  generally  across  the  entire  acquisition  enterprise. 

For  example,  a  key  effort  begun  as  part  of  the  2010  Better  Buying  Power  initiative  was  the 
implementation  of  "should  cost"  based  management,  which  is  fundamental  to  controlling  costs 
through  the  acquisition  life  cycle.  Managers  have  been  tasked  to  establish  "should  cost"  goals 
by  scrutinizing  each  element  of  cost,  then  taking  action  to  purse  these  potential  cost  savings.  In 
FY2012  approximately  one-third  of  all  Acquisition  Category  (ACAT)  l/ll/lll  programs  within  the 
Military  Departments  had  implemented  should-cost;  and  reporting  on  program  should-cost 
execution  savings  in  FY2012  showed  roughly  2  percent  to  3  percent  savings  realized  in  FY2012 
in  these  programs,  as  compared  to  budget  levels.  These  savings  typically  have  been  plowed 
back  into  the  programs  to  gain  higher  productivity. 

Gaps  in  Existing  Analysis 

The  analysis  in  this  report  evolved  through  exploration  of  data  and  analytic  approaches  over 
the  last  2  years.  As  a  result,  earlier  analyses  such  as  the  DOT&E  assessments  of  weapon  system 
performance  do  not  have  the  latest  results  and  thus  could  be  updated  in  subsequent  reports 
when  considered  worthwhile. 

Other  gaps  result  from  either  insufficient  data  (i.e.,  low  sample  count)  or  nonexistent  data  to 
address  a  particular  performance  area.  Even  with  the  large  set  of  MDAP  contracts  we  used,  for 
example,  splitting  between  Military  Departments  in  recent  periods  (e.g.,  2005-2011)  and 
especially  when  also  splitting  by  commodity  type  (as  in  our  analysis  of  acquisition  commands) 
can  result  in  insufficient  data  to  reach  statistical  conclusions. 

Even  more  importantly,  perhaps,  this  stream  of  MDAP  contract  data  requires  a  long  time  before 
we  can  attempt  to  measure  the  effects  of  major  policies  (e.g.,  measuring  whether  cost  growth 
was  lower  after  2009  when  the  Weapon  Systems  Acquisition  Reform  Act  became  law  and  the 
effects  will  take).  Similarly,  at  the  program  level,  1  or  2  years  of  lower  cost  growth  may  or  may 
not  indicate  a  long-term  trend. 

Finally,  the  data  must  have  relevant  variables  or  parameters  at  the  transaction  level  (e.g.,  per 
contract)  that  relate  to  a  topic  of  interest.  If  our  data  only  show,  for  example,  that  an  entire 
Military  Department  follows  a  certain  process  that  makes  it  difficult  to  test  the  implications  of 
that  process  because  the  result  will  correlate  to  the  Department  rather  than  the  process.  We 
need  data  on  cases  in  all  Military  Departments  when  they  followed  the  common  process  and 
when  they  did  not  to  attempt  measurement  of  that  process'  effectiveness. 

In  addition  to  identifying  data,  different  analytic  approaches  and  models  were  created  over  the 
last  2  years.  Some  are  more  sophisticated  than  others,  and  some  merely  partition  the  data  in 
different  ways.  While  multiple  analytic  results  can  be  confusing,  examining  the  data  in 
different  ways  helps  to  validate  findings  and  ensure  that  they  are  robust.  For  example,  different 
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analyses  identified  measurable  improvement  trends  in  controlling  total  contract  cost  growth. 
Although  the  magnitude  of  the  improvements  vary  somewhat,  they  are  similar  and  thus 
support  the  robustness  of  the  finding.  Likewise,  different  analyses  show  that  UCAs  tend  to 
contribute  to  cost  growth  only  on  development  contracts  (although  to  different  degrees  in 
different  sets  of  data).  The  consistency  of  these  observation  helps  to  instill  confidence  in  the 
analysis  and  any  lessons  we  conclude.  Our  efforts  will  continue  to  seek  and  adjust  analytic 
methods  to  the  available  data  and  question. 
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4.  Concluding  Comments 

Taken  together,  measuring  the  performance  of  defense  acquisition  provides  objective, 
quantitative  information  on  our  current  performance.  The  following  insights  provide  some 
broader  perspectives  and  considerations.  These  should  inform  and  enable  stable  improvement 
in  our  overall  acquisition  performance. 

Acquisition  as  an  Endeavor 

Competing  measures  of  performance.  Contextually,  we  note  that  the  time  required  to  acquire 
next-generation  capabilities  is  often  longer  than  the  strategic  threat  and  technology  cycles 
these  capabilities  are  meant  to  address.  Performance  (good  or  bad)  in  planned  defense 
acquisition  is  intertwined  with  cost  and  schedule  implications  from  unplanned  responses  to 
these  external  demands.  This  is  not  an  excuse  for  cost  and  schedule  growth,  but  an  observation 
from  first  principles  that  changing  threats  and  needs  can  add  costs  and  delays  relative  to 
original  baselines  as  ongoing  acquisitions  are  adjusted. 

Acquisition  is  about  risk  management— not  certainties.  Especially  for  major  weapons  systems 
acquisitions  (which  almost  always  involve  research  and  development),  uncertainties  imply  cost, 
schedule,  and  performance  risks  relative  to  early  estimates.  These  risks  diminish  as  we  move 
from  research  to  development  through  production  to  sustainment,  but  their  realization  may 
result  in  cost  and  schedule  growth.  These  risks  also  require  use  of  different  management  tools 
(such  as  the  right  contract  types  and  incentives)  at  different  stages  to  mitigate  risks  and 
motivate  industry  to  achieve  the  lowest  possible  total  price  to  the  government.  We  must 
monitor  and  explain  risks,  but  it  is  important  to  remember  that  developing  technologically 
superior  military  capability  is  not  a  risk-free  endeavor. 

Intelligent  acquisition  is  key.  Analysis  confirms  many  acquisition  fundamentals  and  the  need 
for  informed  judgments.  For  example,  analysis  is  revealing  that  UCAs  can  be  successfully 
employed  in  early  production  but  are  concerns  for  developmental  work.  In  another  example,  no 
single  contract  type  (e.g.,  fixed  price  or  cost-reimbursed)  yields  better  cost  control;  thus,  we 
need  to  rely  on  best  practices  to  select  the  most  appropriate  contract  type  and  incentive 
structure  given  the  maturity,  system  type,  and  acquisition  strategy  for  each  system. 

Beyond  analysis,  experience  also  leads  us  to  assert  that  basic  acquisition  fundamentals  work. 
Premature  contracting  without  a  clear  and  stable  understanding  of  engineering  and  design 
issues  greatly  affects  contract  work  content  stability  and  cost  growth.  In  addition,  first 
principles  indicate  that  concurrent  production  when  designs  are  unstable  can  impose  added 
retrofit  costs  for  early  production  products.  Further  analysis  of  past  performance  will  provide 
an  objective  foundation  for  informing  future  policies  and  acquisition  decisions. 

Programs  with  bad  starts  often  continue  to  have  problems.  Early  work  content  stability  on  a 
contract  predicts  lower  total  cost,  work  content,  and  schedule  growths  (but  not  costs-over- 
target)  on  development  and  early  production  contracts.  Also,  cost  growth  on  development 
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contracts  correlates  strongly  with  cost  growth  on  subsequent  early  production  contracts. 
Independent  analysis  compared  RDT&E  program-level  cost  growth  from  SARs  to  growth  in 
procurement  unit  costs;  regression  analysis  showed  that  each  10  percentage-point  increase  in 
RDT&E  would  be  expected  to  produce  an  increase  of  6  points  in  APUC  (Davis,  2011). 

No  single  contract  type  is  best.  Analysis  of  past  acquisitions  shows  that,  when  controlling  for 
other  factors  that  contribute  to  contract  cost  performance,  contract  type  alone  (e.g.,  fixed  price 
or  cost-reimbursable)  does  not  predict  lower  cost  growth  in  development  or  early  production 
contracts.  This  suggests  that  relying  on  contract  type  alone  to  achieve  better  affordability 
outcomes  will  not  likely  be  successful.  This  does  not  absolve  us  from  the  need  to  carefully 
consider  and  select  the  most  appropriate  contract  type  given  the  maturity,  system  type,  and 
business  strategy  for  each  system. 

On  Measuring  Performance 

Institutional  differences  are  informative.  Performance  differences  between  Military 
Departments,  acquisition  commands,  prime  contractors  (and  even  acquisition  executives) 
reveal  clues  for  improvement  and  potential  areas  ripe  for  analysis.  However,  more  needs  to  be 
done  to  understand  the  causes  of  these  differences  and  identify  transferable  shared  practices. 
For  example,  NAVAIR  controlled  total  contract  cost  growth  better  than  ASC  by  controlling 
contract  work  content  better,  but  how  NAVAIR  controlled  work  content  relative  to  changing 
needs  is  not  yet  clear,  nor  how  much  of  that  added  work  content  was  unavoidable. 

Institutional  differences  are  not  always  decisive.  A  major  shift  in  our  analytic  approach  is  the 
examination  of  institutional  performance  as  opposed  to  the  performance  of  individual 
programs.  Significant  insights  have  been  made,  especially  in  comparing  the  performance  of 
acquisition  commands  when  acquiring  the  same  types  of  systems.  In  many  cases,  however,  the 
analysis  found  that  institutional  differences— while  statistically  significant— are  not  large  nor  do 
they  explain  much  of  the  variation  in  performance.  In  these  cases,  focusing  on  reducing  these 
performance  differences  will  return  little  for  our  effort. 

Performance  relative  to  original  baselines  does  not  guarantee  lowest  end  cost  to  the 
taxpayers.  Baselines  at  Milestone  B  are  static  yet  help  to  control  appetites  and  provide  valuable 
triggers  to  explain  substantial  changes.  Original  baselines  at  Milestone  B  provide  a  useful 
reference  point  for  subsequent  cost  and  schedule  performance,  but  developmental 
uncertainties,  changing  operational  needs,  and  other  mechanisms  for  controlling  costs  make 
such  baselines  somewhat  arbitrary  and  misleading  if  used  as  absolute  reference  points. 
Techniques  to  control  immediate  costs  and  moderate  capability  appetite  (e.g.,  block  upgrades; 
taking  content  out  of  a  program;  postponing  fixes  until  sustainment)  can  control  cost  relative  to 
certain  baselines  yet  delay  capabilities  and  push  added  costs  into  (monitored  or  unmonitored) 
future  activities.  Still,  baselines  are  useful  tools  to  help  monitor  costs  and  ground  agreements 
within  the  Department  and  with  Congress  on  what  we  are  acquiring  and  at  what  cost. 
Moreover,  in-depth  monitoring  of  cost  growth  at  different  levels  and  in  different  ways  helps  to 


110 


Performance  of  the  Defense  Acquisition  System ,  2013 


increase  transparency  and  the  Department's  cost  consciousness  in  balancing  timely  acquisition 
of  warfighter  capabilities  while  being  responsible  to  the  taxpayers. 

Contract  measures  help  inform  causes  of  cost  and  schedule  growth.  For  example,  total 
contract  cost  growth  can  be  split  between  work  content  purposely  added  to  a  contract  and 
costs-over-target.  Contract  work  content  growth  dominates  total  cost  growth  statistically,  but 
costs-over-target  also  are  significant  and  worrisome.  Cost-over-target  reflects  poor 
performance,  poor  estimation,  or  faulty  framing  assumptions  and  generally  is  bad.  Work 
content  growth,  on  the  other  hand,  may  (in  part)  reflect  normal,  necessary  additions  that 
resolve  and  reduce  problems  from  technical  and  engineering  uncertainties.  Still,  requirements 
creep  can  be  a  factor  in  work  content  growth  and  needs  to  be  recognized  and  subject  to 
affordability  constraints. 

Current  Acquisition  Performance 

Current  execution  performance  is  mixed.  Performance  across  all  MDAPs  at  the  program  level  is 
highly  skewed,  with  a  few  outliers  with  extremely  high  cost  growth,  a  large  population  with 
moderate  cost  growth,  and  a  small  number  with  negative  or  negligible  cost  growth.  Over  the 
last  10  years  analyzed,  median  program  RDT&E  cost  growths  have  ranged  from  5  percent  to  18 
percent  (inflation  adjusted),  while  median  quantity-adjusted  program  unit  costs  have  ranged 
from  3  percent  to  9  percent  (inflation  adjusted),  measured  against  original  Milestone  B 
baselines.  While  unacceptable,  problems  on  some  programs  such  as  the  F-35  mask  the  fact  that 
some  DoD  programs  perform  relatively  well,  raising  the  question  of  how  the  Department  can 
be  more  consistent  in  its  performance. 

Performance  at  the  MDAP  contract  level  also  is  highly  skewed,  but  the  cost  growth  is  higher 
than  measured  at  the  program  level.  Twenty-year  median  total  cost  growth  has  been  about  31 
percent  (unadjusted  for  inflation)  on  development  contracts  and  10  percent  (unadjusted  for 
inflation)  on  early  production  contracts,  measured  against  original  contract  cost  targets. 
Between  organizations,  performance  varies  significantly  between  Military  Departments, 
acquisition  commands,  and  commercial  companies  in  measurable  ways,  suggesting  questions  of 
why  and  whether  lessons  learned  can  be  shared  to  improve  performance.  While  not  excusing 
poor  performance,  large  commercial  projects  also  exhibit  significant  and  skewed  cost  growths 
similar  to  those  in  the  DoD.  This  helps  to  support  the  hypothesis  that  technical  and  engineering 
risks  on  complex,  state-of-the-art  systems  are  somewhat  unpredictable,  contributing  to  cost 
and  schedule  growth. 

Performance  is  improving.  Some  program  and  organizational  performances  are  better  in  some 
measures,  but  more  progress  is  needed.  Very  recent  data  show  statistically  significant 
improvement,  but  only  further  analysis  will  tell  if  these  trends  continue  into  the  future.  For 
example,  comparing  the  last  2  decades,  the  Army  and  Air  Force  have  reduced  total  cost  growth 
on  contracts,  and  the  Army  has  reduced  contract  costs-over-target.  The  Air  Forcealso  has 
lowered  contract  schedule  growth.  Despite  such  trends,  the  magnitude  of  absolute 
performance  issues  leaves  considerable  room  for  additional  improvement. 
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Affordability  will  be  a  major  challenge  in  the  next  few  years.  While  most  cost  growth 
measures  are  improving,  median  cost  growth  across  the  MDAP  portfolio  is  not  zero  and  will 
likely  lead  to  near-term  affordability  challenges  given  flat  or  declining  fiscal  resources. 
Achieving  affordability  by  removing  capability  or  requirements  may  help  reduce  cost  growth, 
and  SAR  reports  at  the  program  level,  and  work  content  measures  at  the  contract  level  should 
reveal  whether  such  actions  are  being  taken. 
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A.  Abbreviations 


AAC-Air  Armament  Center  [Air  Force] 

AEHF-Advanced  Extremely  High  Frequency 

AFATDS-Advanced  Field  Artillery  Tactical 
Data  System 

AFMC-Air  Force  Materiel  Command 

AMC-Army  Material  Command 

AMCOM-Aviation  and  Missile  Command 
[Army] 

AMP -Avionics  Modernization  Program 

AoA-Analysis  of  Alternatives 

APUC  —  Average  Procurement  Unit  Cost 

ARH-Armed  Reconnaissance  Helicopter 

ASC-Aeronautical  Systems  Center  [Air 
Force] 

AT&L-Acquisition,  Technology,  and  Logistics 

BAMS-Broad  Area  Maritime  Surveillance 

CBB— Contract  Budget  Base 

CEDD-Continuing  Engineering  Design/ 
Development 

CJR-Cobra  Judy  Replacement 
CPAF— Cost  Plus  Award  Fee 
CPFF— Cost  Plus  Fixed  Fee 
CPIF— Cost  Plus  Incentive  Fee 
DDS-Dry  Dock  Shelter 
DoD-Department  of  Defense 


DOT&E— Director,  Operational  Test  and 
Evaluation 

DSUP-Defense  System  Upgrade  Program 

DVH-Double-V  Hull 

EFV-Expeditionary  Fighting  Vehicle 

EMD— Engineering,  Manufacturing  and 
Development 

FAB-T-Family  of  Advanced  Beyond-line-of- 
sight  Terminals 

FCS-Future  Combat  Systems 

FPIF— Fixed  Price  Incentive  Firm 

FFP— Firm  Fixed  Price 

FY— Fiscal  Year 

Granite  Sentry-aircraft  tracking  system 

GBR-Ground  Based  Radar 

GBS-Global  Broadcast  Service 

GCCS-Global  Command  and  Control  System 

GCSS-Global  Combat  Support  System 

GPS-Global  Positioning  System 

lAMD-Integrated  Air  and  Missile  Defense 

JASSM-Joint  Air-to-Surface  Stand-off 
Missile 

JDAM-Joint  Direct  Attack  Munitions 

JHSV-Joint  High  Speed  Vessel 

JLENS-Joint  Land  Attack  Cruise  Missile 
Defense  Elevated  Netted  Sensor  System 


113 


Performance  of  the  Defense  Acquisition  System ,  2013 


JM&LLCMC— Joint  Munitions  and  Lethality 
Life  Cycle  Management  Command 

JSF— Joint  Strike  Fighter 

JSOW-Joint  Stand-off  Weapon 

JSTARS-Joint  Surveillance  Target  Attack 
Radar  System 

JTRS-Joint  Tactical  Radio  System 

KPP— Key  Performance  Parameter 

LADS-Laser  Area  Defense  System 

LRIP-Low-Rate  Initial  Production 

MDA-Milestone  Decision  Authority 

MDAP— Major  Defense  Acquisition  Program 

MDD-Materiel  Development  Decision 

MEADS-Medium  Extended  Air  Defense 
System 

MIDS-LVT-Multi-functional  Information 
Distribution  System-Low  Volume  Terminal 

MRAP-Mine  Resistant  Ambush  Protected 

MS-Milestone 

MUOS-Mobile  User  Objective  System 
NAVAIR-Naval  Air  Systems  Command 


NAVSEA— Naval  Sea  Systems  Command 

PAC-3-Patriot  Advanced  Capability-  3 

PARCA— Performance  Assessments  and 
Root  Cause  Analysis 

PAUC— Program  Acquisition  Unit  Cost 

PM  EAC— Program  Manager  Estimate  at 
Completion 

RDT&E— Research  Development  Test  and 
Evaluation 

SAE-Service  Acquisition  Executive 

SAR— Selected  Acquisition  Report 

SBIRS-Space-based  Infrared  System 

SCAMP-Single-channel  Anti-jam  Man- 
portable 

TMD-Theater  Missile  Defense 

UCA— Undefinitized  Contract  Action 

U.S.C.— United  States  Code 

USD-Under  Secretary  of  Defense 

WIN-T-Warfighter  Information  Network - 
Tactical 
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